问题描述
限时送ChatGPT账号..我使用 ANTLRWorks 创建了以下词法分析器.(另见 http://bkiers.blogspot/2011/03/2-introduction-to-antlr.html#intro )
I created the following Lexer using ANTLRWorks. ( See also http://bkiers.blogspot/2011/03/2-introduction-to-antlr.html#intro )
// CSVLexer.g
lexer grammar CSVLexer;
@lexer::header {
package graphica.parsers;
}
Comma
: ','
;
LineBreak
: '\r'? '\n'
| '\r'
;
SimpleValue
: ~(',' | '\r' | '\n' | '"')+
;
QuotedValue
: '"' ('""' | ~'"')* '"'
;
我使用以下 Java 类来测试 Lexer.
I used the following Java class to test the Lexer.
/**
*
* @author Nilo
*/
import org.antlr.runtime.*;
public class CSVLexerTest {
public static void main(String[] args) throws Exception {
// the input source
String source =
"val1, value2, value3, value3.2" + "\n"
+ "\"line\nbreak\",ABAbb,end";
// create an instance of the lexer
CSVLexer lexer = new CSVLexer(new ANTLRStringStream(source));
// wrap a token-stream around the lexer
CommonTokenStream tokens = new CommonTokenStream(lexer);
// traverse the tokens and print them to see if the correct tokens are created
// tokens.toString();
int n = 1;
for (Object o : tokens.getTokens()) {
CommonToken token = (CommonToken) o;
System.out.println("token(" + n + ") = " + token.getText().replace("\n", "\\n"));
n++;
}
}
}
上面的类(来自同一个教程)不产生任何输出.然而,如果我在令牌循环之前插入一个 tokens.toString() ,那么输出将按预期打印.
The class above ( from the same tutorial ) does NOT produce any output. If I however insert a tokens.toString() prior to the token loop then output is printed as expected.
注意:我在带有 JDK 1.7/64 位的 Windows 7 上使用 ANTLWorks 1.4.3、ANTLR 3.4
Note: I use ANTLWorks 1.4.3, ANTLR 3.4, on Windows 7 with JDK 1.7/64bit
问题:我不明白这一点.请解释.应该有一种方法可以在没有 tokens.toString() 的情况下使其正常工作
推荐答案
CommonTokenStream
extends BufferedTokenStream
有一个 List
.但是这个 getTokens()
时返回的令牌List
仅在特定时间填充.在 3.3 和 3.4 中,它不会不会发生在 getTokens()
之后,其中 3.2 会 填充 tokens
列表.>
ANTLR 3.2(及之前)
CommonTokenStream
extends BufferedTokenStream
which has a List<Token> tokens
that is returned when one calls getTokens()
. But this List<Token> tokens
only gets filled at certain times. In 3.3 and 3.4 it does not happen after getTokens()
where 3.2 does fill the tokens
list.
public List getTokens() {
if ( p == -1 ) {
fillBuffer();
}
return tokens;
}
protected void fillBuffer() {
// fill `tokens`
}
ANTLR 3.3(及之后)
public List getTokens() {
return tokens;
}
public void fill() {
// fill `tokens`
}
请注意 3.2 的 fill 方法是如何受保护的,并且在 3.3+ 中它是公共的,因此以下有效:
Notice how 3.2's fill method is protected and in 3.3+ it is public, so the following works:
import org.antlr.runtime.*;
public class CSVLexerTest {
public static void main(String[] args) throws Exception {
// the input source
String source =
"val1, value2, value3, value3.2" + "\n" +
"\"line\nbreak\",ABAbb,end";
// create an instance of the lexer
CSVLexer lexer = new CSVLexer(new ANTLRStringStream(source));
// wrap a token-stream around the lexer and fill the tokens-list
CommonTokenStream tokens = new CommonTokenStream(lexer);
tokens.fill();
// traverse the tokens and print them to see if the correct tokens are created
// tokens.toString();
int n = 1;
for (Object o : tokens.getTokens()) {
CommonToken token = (CommonToken) o;
System.out.println("token(" + n + ") = " + token.getText().replace("\n", "\\n"));
n++;
}
}
}
产生输出:
token(1) = val1
token(2) = ,
token(3) = value2
token(4) = ,
token(5) = value3
token(6) = ,
token(7) = value3.2
token(8) = \n
token(9) = "line\nbreak"
token(10) = ,
token(11) = ABAbb
token(12) = ,
token(13) = end
token(14) = <EOF>
这篇关于ANTLR API 问题;提供示例+解决方法;需要解释的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
更多推荐
[db:关键词]
发布评论