Sqoop和Java 7

本文介绍了Sqoop和Java 7的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我试图使用sqoop将一个MySQL表导入HDFS。我正在使用JDK 1.7.0_45和CDH4.4。我实际上使用了cloudera的预建VM，不过我将JDK更改为1.7，因为我想使用pydev插件进行eclipse。我的sqoop版本是1.4.3-cdh4.4.0。

当我运行sqoop时，出现以下异常：

错误：商品：不支持major.minor版本51.0

过去我看到过这个错误： 1.编译为java 7 2.用java 6运行应用程序。

但这不是我现在所做的。我相信我的sqoop版本被编译为java 6，而且我正在使用java 7来运行它，这应该是非常好的。我想可能hadoop正在用JDK 6启动映射程序，我不知道如何改变它。我浏览了mapred配置文档，并没有看到设置Java版本以用于map任务的任何方式。

以下是相关的控制台输出：

[cloudera @ localhost〜] $ echo $ JAVA_HOME / usr / java / latest [cloudera @ localhost〜 ] $ java -version java版本1.7.0_45 Java™SE运行时环境（build 1.7.0_45-b18） Java HotSpot™64位服务器虚拟机构建24.45-b08，混合模式） [cloudera @ localhost〜] $ sqoop版本 Sqoop 1.4.3-cdh4.4.0 git commit id 2cefe4939fd464ba11ef63e81f46bbaabf1f5bc6 由jenkins在星期二编译Sep 3 20:41:55 PDT 2013 [cloudera @ localhost〜] $ hadoop version Hadoop 2.0.0-cdh4.4.0 Subversion file：/// data / 1 / jenkins / workspace / generic-package-rhel64-6-0 / topdir / BUILD / hadoop-2.0.0-cdh4.4.0 / src / hadoop-common-project / hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79 编译jenkins于星期二九月3 19:33:17 PDT 2013 来自智慧h checksum ac7e170aa709b3ace13dc5f775487180 此命令使用/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar [cloudera @ localhost〜] $ cat mysqooper.sh ＃！/ bin / bash sqoop import -m 1 --connect jdbc：mysql：// localhost / $ 1 \ --username root --table $ 2 --target-dir $ 3 [cloudera @ localhost〜] $ ./mysqooper.sh云商品/用户/ cloudera /商品/ csv / sqooped 14/01/16 16:45:10信息manager.MySQLManager：准备使用MySQL流式结果集。 14/01/16 16:45:10 INFO tool.CodeGenTool：开始代码生成 14/01/16 16:45:11 INFO manager.SqlManager：执行SQL语句：SELECT t。* FROM `商品`as t LIMIT 1 14/01/16 16:45:11 INFO manager.SqlManager：执行SQL语句：SELECT t。* FROM`commodity` as t LIMIT 1 14/01 / 16 16:45:11 INFO orm.CompilationManager：HADOOP_MAPRED_HOME是/usr/lib/hadoop-0.20-mapreduce 14/01/16 16:45:11信息orm.CompilationManager：找到hadoop核心jar：/ usr /lib/hadoop-0.20-mapreduce/hadoop-core.jar 注意：/tmp/sqoop-cloudera/compile/f75bf6f8829e8eff302db41b01f6796a/commodity.java使用或覆盖弃用的API。 注意：使用-Xlint：deprecation重新编译以获取详细信息。 14/01/16 16:45:15 INFO orm.CompilationManager：编写jar文件：/tmp/sqoop-cloudera/compile/f75bf6f8829e8eff302db41b01f6796a/commodity.jar 14/01/16 16:45： 15 WARN manager.MySQLManager：看起来你是从mysql导入的。 14/01/16 16:45:15 WARN manager.MySQLManager：这个传输可以更快！使用--direct 14/01/16 16:45:15 WARN manager.MySQLManager：选项来执行特定于MySQL的快速路径。 14/01/16 16:45:15信息manager.MySQLManager：将零DATETIME行为设置为convertToNull（mysql） 14/01/16 16:45:15信息mapreduce.ImportJobBase：开始导入商品 14/01/16 16:45:17警告mapred.JobClient：使用GenericOptionsParser解析参数。应用程序应该实现相同的工具。 14/01/16 16:45:20信息mapred.JobClient：正在运行的作业：job_201401161614_0001 14/01/16 16:45:21信息mapred.JobClient：map 0％reduce 0％ 14/01/16 16:45:38信息mapred.JobClient：任务ID：attempt_201401161614_0001_m_000000_0，状态：FAILED 错误：商品：不受支持major.minor版本51.0 14/01/16 16： 45:46信息mapred.JobClient：任务ID：attempt_201401161614_0001_m_000000_1，状态：FAILED 错误：商品：不受支持的major.minor版本51.0 14/01/16 16:45:54信息mapred.JobClient：Task Id：attempt_201401161614_0001_m_000000_2，状态：FAILED 错误：商品：不受支持major.minor版本51.0 14/01/16 16:46:07信息mapred.JobClient：作业完成：job_201401161614_0001 14 / 01/16 16:46:07信息mapred.JobClient：计数器：6 14/01/16 16:46:07信息mapred.JobClient：作业计数器 14/01/16 16:46： 07信息mapred.JobClient：失败的地图任务= 1 14/01/16 16:46:07信息mapred.JobClient：启动的地图任务= 4 14/01/16 16:46:07信息mapred.JobClient：占用插槽中所有地图花费的总时间（ms）= 23048 14/01/16 16:46:07 INFO mapred。 JobClient：占用插槽中所有缩减花费的总时间（毫秒）= 0 14/01/16 16:46:07信息mapred.JobClient：预留插槽后等待的所有地图花费的总时间（毫秒）= 0 14/01/16 16:46:07信息mapred.JobClient：所有减少等待时间（ms）= 0 14/01/16 16:46:07警告mapreduce .Counters：组FileSystemCounters已弃用。使用org.apache.hadoop.mapreduce.FileSystemCounter而不是 14/01/16 16:46:07 INFO mapreduce.ImportJobBase：在51.0252秒（0字节/秒）内传输0个字节 14/01 / 16 16:46:07 WARN mapreduce.Counters：Group org.apache.hadoop.mapred.Task $ Counter已弃用。使用org.apache.hadoop.mapreduce.TaskCounter而不是 14/01/16 16:46:07 INFO mapreduce.ImportJobBase：检索到0条记录。 14/01/16 16:46:07错误tool.ImportTool：导入期间出错：导入作业失败！

我试着用JDK 1.6运行它，但它真的不想切换回每次我需要使用sqoop。

有人知道我需要改变吗？

解决方案

我认为问题的根本原因在于您的Hadoop发行版仍然在JDK6上运行，而不是JDK7，因为您认为是这样。

Sqoop进程将生成使用当前使用的JDK编译的Java代码。因此，如果您在JDK7上执行Sqoop，它将使用此JDK7生成并编译代码。生成的代码随后会作为mapreduce作业的一部分提交给您的hadoop集群。因此，如果您在JDK7上运行Sqoop时收到不支持的major.minr异常，很可能您的Hadoop集群正在JDK6上运行。

我强烈建议在jinfo上调用您的hadoop deamons来验证它们正在运行的JDK。

I'm trying to use sqoop to import a MySQL table into HDFS. I'm using JDK 1.7.0_45 and CDH4.4. I'm actually using cloudera's pre-built VM, except I changed the JDK to 1.7 because I wanted to use the pydev plugin for eclipse. My sqoop version is 1.4.3-cdh4.4.0.

When I run sqoop I get this exception:

Error: commodity : Unsupported major.minor version 51.0

I have seen this error in the past when I did this: 1. compiled to java 7 2. ran an application with java 6.

but that is not what I am doing this time. I believe my sqoop version was compiled to java 6, and I'm running it with java 7, which should be perfectly fine. I think maybe hadoop is launching mapper processes with JDK 6, I have no idea how to change that. I skimmed through the mapred configuration documentation, and did not see any way to set the java version to use for map tasks.

Here is the relevant console output:

[cloudera@localhost ~]$ echo $JAVA_HOME /usr/java/latest [cloudera@localhost ~]$ java -version java version "1.7.0_45" Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) [cloudera@localhost ~]$ sqoop version Sqoop 1.4.3-cdh4.4.0 git commit id 2cefe4939fd464ba11ef63e81f46bbaabf1f5bc6 Compiled by jenkins on Tue Sep 3 20:41:55 PDT 2013 [cloudera@localhost ~]$ hadoop version Hadoop 2.0.0-cdh4.4.0 Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79 Compiled by jenkins on Tue Sep 3 19:33:17 PDT 2013 From source with checksum ac7e170aa709b3ace13dc5f775487180 This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar [cloudera@localhost ~]$ cat mysqooper.sh #!/bin/bash sqoop import -m 1 --connect jdbc:mysql://localhost/$1 \ --username root --table $2 --target-dir $3 [cloudera@localhost ~]$ ./mysqooper.sh cloud commodity /user/cloudera/commodity/csv/sqooped 14/01/16 16:45:10 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 14/01/16 16:45:10 INFO tool.CodeGenTool: Beginning code generation 14/01/16 16:45:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `commodity` AS t LIMIT 1 14/01/16 16:45:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `commodity` AS t LIMIT 1 14/01/16 16:45:11 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-0.20-mapreduce 14/01/16 16:45:11 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar Note: /tmp/sqoop-cloudera/compile/f75bf6f8829e8eff302db41b01f6796a/commodity.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 14/01/16 16:45:15 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/f75bf6f8829e8eff302db41b01f6796a/commodity.jar 14/01/16 16:45:15 WARN manager.MySQLManager: It looks like you are importing from mysql. 14/01/16 16:45:15 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 14/01/16 16:45:15 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 14/01/16 16:45:15 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 14/01/16 16:45:15 INFO mapreduce.ImportJobBase: Beginning import of commodity 14/01/16 16:45:17 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 14/01/16 16:45:20 INFO mapred.JobClient: Running job: job_201401161614_0001 14/01/16 16:45:21 INFO mapred.JobClient: map 0% reduce 0% 14/01/16 16:45:38 INFO mapred.JobClient: Task Id : attempt_201401161614_0001_m_000000_0, Status : FAILED Error: commodity : Unsupported major.minor version 51.0 14/01/16 16:45:46 INFO mapred.JobClient: Task Id : attempt_201401161614_0001_m_000000_1, Status : FAILED Error: commodity : Unsupported major.minor version 51.0 14/01/16 16:45:54 INFO mapred.JobClient: Task Id : attempt_201401161614_0001_m_000000_2, Status : FAILED Error: commodity : Unsupported major.minor version 51.0 14/01/16 16:46:07 INFO mapred.JobClient: Job complete: job_201401161614_0001 14/01/16 16:46:07 INFO mapred.JobClient: Counters: 6 14/01/16 16:46:07 INFO mapred.JobClient: Job Counters 14/01/16 16:46:07 INFO mapred.JobClient: Failed map tasks=1 14/01/16 16:46:07 INFO mapred.JobClient: Launched map tasks=4 14/01/16 16:46:07 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=23048 14/01/16 16:46:07 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0 14/01/16 16:46:07 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 14/01/16 16:46:07 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 14/01/16 16:46:07 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 14/01/16 16:46:07 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 51.0252 seconds (0 bytes/sec) 14/01/16 16:46:07 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 14/01/16 16:46:07 INFO mapreduce.ImportJobBase: Retrieved 0 records. 14/01/16 16:46:07 ERROR tool.ImportTool: Error during import: Import job failed!

I tried running with JDK 1.6 and it works, but I really don't want to switch back to that every time I need to use sqoop.

Does anybody know what I need to change?

解决方案

I belive that root cause of your problem is that your Hadoop distribution is still running on JDK6 and not JDK7 as you believe so.

Sqoop process will generate Java code that is compiled with currently used JDK. Therefore if you execute Sqoop on JDK7, it will generate and compile code with this JDK7. The generated code is then submitted to your hadoop cluster as a part of mapreduce job. Therefore if you are getting this unsupported major.minr exception while running Sqoop on JDK7 is very likely that your Hadoop cluster is running on JDK6.

I would strongly suggest calling jinfo on your hadoop deamons to verify which JDK they are running on.