使用Pig访问Hbase(Access Hbase with Pig)

编程入门行业动态更新时间:2024-10-13 02:13:44

我试图使用Pig（0.15.0）从HBase（1.1.2）加载数据。我试过注册过多的JAR而没有任何成功。我可以用Java访问它但不能用Pig访问它。下面我注册了所有的JAR。前五个，然后是lib/*.jar then target/alternateLocation/*.jar是我的Java代码使用的lib/*.jar then target/alternateLocation/*.jar 。所有东西都安装在我的本地PC上，并且HBase在HDFS上运行。

--REGISTER /home/halmousli/hbase-1.1.2/lib/*.jar --REGISTER /home/halmousli/mygit/Hbase/apachehbase/target/alternateLocation/*.jar set hbase.zookeeper.quorum 'localhost' data = LOAD 'hbase://emp' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('professional:designation', '-loadKey true') as (id:bytearray,prof:CHARARRAY); DUMP data;

猪脚本无法验证：java.lang.RuntimeException：无法实例化'org.apache.pig.backend.hadoop.hbase.HBaseStorage'参数为'[professional：designation，-loadKey true]'解析失败：Pig脚本失败解析：pig脚本无法验证：java.lang.RuntimeException：无法在org.apache实例化带有参数'[professional：designation，-loadKey true]'的'org.apache.pig.backend.hadoop.hbase.HBaseStorage'在org.apache.pig.PigServer上的.pig.parser.QueryParserDriver.parse（QueryParserDriver.java:199）。org.apache.pig.PigServer $ Graph.access $ Pig（PigServer .PigServer $ Graph.parseQuery（PigServer.java:1735） java：1443）at org.apache.pig.PigServer.parseAndBuild（PigServer.java:387）at org.apache.pig.PigServer.executeBatch（PigServer.java:412）at org.apache.pig.PigServer.executeBatch（PigServer .java：398）at org.apache.pig.tools.grunt.GruntParser.executeBatch（GruntParser.java:171）at org.apache.pig.tools.grunt.GruntParser.processDump（GruntParser.java:749）at org。 apache.pig.tools.pig script.parser.PigScriptParser.parse（PigScriptParser.java:376）at org.apache.pig.tools.grunt.GruntParser.parseStopOnError（GruntParser.java:230）at org.apache.pig.tools.grunt.GruntParser.parseStopOnError（ GruntParser.java:205）在org.apache.pig.tools.grunt.Grunt.exec（Grunt.java:81）在org.apache.pig.Main.run（Main.java:631）在org.apache.pig .Main.main（Main.java:177）at sun.reflect.NativeMethodAccessorImpl.invoke0（Native Method）at sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java ：43）在org.apache.hadoop.util.RunJar的org.apache.hadoop.util.RunJar.run（RunJar.java:221）处的java.lang.reflect.Method.invoke（Method.java:606）。 main（RunJar.java:136）引起：/home/halmousli/mygit/Pig/mavenpigtester/src/main/resources/hbaseGetter.pig，第21行，第7列>猪脚本无法验证：java.lang.RuntimeException：无法实例化'org.apache.pig.backend.hadoop.hbase.HBaseStorage'wi 参数org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp（LogicalPlanBuilder.java:897）在org.apache.pig.parser.LogicalPlanGenerator.load_clause（LogicalPlanGenerator.java:3568）上的参数'[professional：designation，-loadKey true]' ）在org.apache.pig.parser.LogicalPlanGenerator.op_clause（LogicalPlanGenerator.java:1625）处，位于org.apache.pig.parser.LogicalPlanGenerator.general_statement（LogicalPlanGenerator.java:1102）处，位于org.apache.pig.parser.LogicalPlanGenerator。语句（LogicalPlanGenerator.java:560）在org.apache.pig.parser.LogicalPlanGenerator.query（LogicalPlanGenerator.java:421）at org.apache.pig.parser.QueryParserDriver.parse（QueryParserDriver.java:191）... 19更多原因：java.lang.RuntimeException：无法在org.apache.pig.impl处使用参数'[professional：designation，-loadKey true]'实例化'org.apache.pig.backend.hadoop.hbase.HBaseStorage'。 PigContext.instantiateFuncFromSpec（PigContext.java:772）at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp（LogicalPlanBuilder。 java：885）... 25更多引起：java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0（Native Method）at sun.reflect.NativeConstructorAccessorImpl.newInstance（NativeConstructorAccessorImpl.java:57）at sun.reflect .DelegatingConstructorAccessorImpl.newInstance（DelegatingConstructorAccessorImpl.java:45）at java.lang.reflect.Constructor.newInstance（Constructor.java:526）at org.apache.pig.impl.PigContext.instantiateFuncFromSpec（PigContext.java:740）... 26更多引起：java.lang.NoSuchMethodError：org.apache.hadoop.hbase.client.Scan.setCacheBlocks（Z）V at org.apache.pig.backend.hadoop.hbase.HBaseStorage.initScan（HBaseStorage.java:427）at org.apache.pig.backend.hadoop.hbase.HBaseStorage。（HBaseStorage.java:368）... 31更多2015-12-30 11：19：14,794 [main] ERROR org.apache.pig.tools.grunt。 Grunt - 错误1200：猪脚本无法解析：猪脚本无法验证：java.lang.RuntimeException：无法实例化'org.apache.pig.backend.hadoop.hbase.HBaseStorage'参数'[professional：designation， - loadKey true]'日志文件的详细信息：/home/halmousli/hadoop-2.6.0/sbin/pig_1451492353570.log

我不知道为什么我看到： java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks 。在我的Java代码中，我能够找到该功能并运行它，但不是在Pig中，而是使用相同的罐子。

Scan scan = new Scan(); scan.setCacheBlocks(true);

任何建议来解决这个问题或以其他方式从猪访问HBase？谢谢。

I'm trying to load data from HBase (1.1.2) using Pig (0.15.0). I've tried registering too many JARs without any success. I'm able to access it with Java but not with Pig. Below I've registered all JARs. the first five, then lib/*.jar then target/alternateLocation/*.jar which my Java code was using. Everything is installed on my local PC and HBase is running on HDFS.

pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' Failed to parse: Pig script failed to parse: pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735) at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1443) at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387) at org.apache.pig.PigServer.executeBatch(PigServer.java:412) at org.apache.pig.PigServer.executeBatch(PigServer.java:398) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:749) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:631) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: /home/halmousli/mygit/Pig/mavenpigtester/src/main/resources/hbaseGetter.pig, line 21, column 7> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 19 more Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885) ... 25 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740) ... 26 more Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V at org.apache.pig.backend.hadoop.hbase.HBaseStorage.initScan(HBaseStorage.java:427) at org.apache.pig.backend.hadoop.hbase.HBaseStorage.(HBaseStorage.java:368) ... 31 more 2015-12-30 11:19:14,794 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' Details at logfile: /home/halmousli/hadoop-2.6.0/sbin/pig_1451492353570.log

I do not know why I am seeing caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks. In my Java code, I am able to find that function and run it but not in Pig and I am using the same jars.

Scan scan = new Scan(); scan.setCacheBlocks(true);

Any suggestion to solve this problem or to other way to access HBase from Pig?. Thanks.

最满意答案

上述错误的原因是我的hadoop版本和hbase不兼容。我安装了hbase-0.98.16-hadoop2，猪代码运行时没有错误。

通常当你看到java.lang.NoSuchMethodError时，它可能是版本之间不兼容的一个指示。

The reason for the error above was incompatibility between my hadoop version and hbase. I installed hbase-0.98.16-hadoop2 and the pig code is running without errors.

Generally when you see java.lang.NoSuchMethodError then it can be an indication of incompatibility between versions.

更多推荐

本文发布于:2023-08-05 21:13:00，感谢您对本站的认可！

本文链接:https://www.elefans.com/category/jswz/34/1437694.html