我有一个ID终结者
ID : ([A-Z_]|'\u0100'..'\uFFFE') ([A-Z_0-9]|'\u0100'..'\uFFFE')*;和一个.txt示例文件来解析
均60:=MA(C,60);我生成了Java和Python2目标,并分别针对样本文件进行测试。 Java目标可以解析这个文件。 但是Python2目标不能。 它会token recognition error at: '均'抛出token recognition error at: '均' 。 我测试了其他有效输入的Python2目标,除了包含unicode字符的所有作品。 我错过了什么或python目标不支持unicode解析。
java的
mkdir -p java java -jar /usr/local/lib/antlr-4.5.3-complete.jar TDX.g4 -o ./java cd ./java javac TDX*.java java org.antlr.v4.gui.TestRig TDX prog -gui ../samples/1.txtpython目标生成命令
java -jar /usr/local/lib/antlr-4.5.3-complete.jar -Dlanguage=Python2 TDX.g4 -o ./tdx_py/antlrgen -visitorpython代码
import sys from antlr4 import * from tdx_py.antlrgen import TDXLexer, TDXParser def executefile(file): input = FileStream(file, encoding='utf-8') lexer = TDXLexer(input) stream = CommonTokenStream(lexer) parser = TDXParser(stream) tree = parser.prog() if __name__ == '__main__': executefile(sys.argv[1])I have a ID terminator
ID : ([A-Z_]|'\u0100'..'\uFFFE') ([A-Z_0-9]|'\u0100'..'\uFFFE')*;and a .txt sample file to parse
均60:=MA(C,60);I generated Java and Python2 target and test each against sample file respectively. Java target can parse this file. But Python2 target can't. It throws token recognition error at: '均'. And I tested Python2 target against other valid inputs, all works except which contains unicode characters. Did I miss something or python target don't support unicode parsing.
java
mkdir -p java java -jar /usr/local/lib/antlr-4.5.3-complete.jar TDX.g4 -o ./java cd ./java javac TDX*.java java org.antlr.v4.gui.TestRig TDX prog -gui ../samples/1.txtpython target generating command
java -jar /usr/local/lib/antlr-4.5.3-complete.jar -Dlanguage=Python2 TDX.g4 -o ./tdx_py/antlrgen -visitorpython code
import sys from antlr4 import * from tdx_py.antlrgen import TDXLexer, TDXParser def executefile(file): input = FileStream(file, encoding='utf-8') lexer = TDXLexer(input) stream = CommonTokenStream(lexer) parser = TDXParser(stream) tree = parser.prog() if __name__ == '__main__': executefile(sys.argv[1])最满意答案
这是ANTLR4的一个错误。 参考https://github.com/antlr/antlr4/issues/1925
This is a bug of ANTLR4. Reference https://github.com/antlr/antlr4/issues/1925
更多推荐
发布评论