hadoop 2.6.0编译native library"/>
mac下hadoop 2.6.0编译native library
- 本文中的所有路径请根据个人情况修改。
- 编译好的native library见个人资源:【】
一、为什么要编译native library
mac单机模式安装Hadoop后启动,报错:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable。
有兴趣的可以自己编译下,因为网上很多的native library资源和方法都是不可用的,自己在本地编译也是官网推荐的。
二、问题原因
在官网上有说明:
The pre-built 32-bit i386-Linux native hadoop library is available as part of the hadoop distribution and is located in the lib/native directory. You can download the hadoop distribution from Hadoop Common Releases.
在官网下载的Hadoop版本中自带的native library 只针对32位Linux。
The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.
官网已明确说明预编译的native library 不适用于mac os x。所以这也是必须要在mac下编译native library的充要理由。
详细说明请参见.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html
三、编译环境说明
- mac os x:10.10.4
- jdk:java version “1.7.0_80”
- hadoop:2.6.0-src
四、编译前期准备
编译开始前需要在mac下安装以下软件:
1.安装brew
类似Ubuntu中的apt软件包管理工具,用于安装缺少的软件包,这里用于安装cmake工具:brew install cmake。
2.安装cmake
没有版本要求。
3.安装protoc
版本必须是2.5.0,否则编译失败,不能用brew install protobuf的方式安装,因为该方式安装的版本不一定是2.5.0。
protocbuf 2.5.0的源码详见个人资源【】,需要自己手动编译,其实也比较简单,步骤如下:
(1)设置编译目录:
./configure --prefix=/User/King-pan/software/tools/protobuf
其中/User/King-pan/software/tools/protobuf 为自己设定的编译安装目录。
(2)安装:
make
make install
(3)配置环境变量:
sudo vi .bash_profile
(4)添加配置文件:
export PROTOBUF=/Users/King-pan/software/tools/protobuf
export PATH=$PROTOBUF/bin:$PATH
(5)测试:
protoc --version
4.安装maven
本人编译时使用的版本:Apache Maven 3.3.3。
五、最后准备
在编译过程中会出现以下报错:
Exception in thread “main” java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()
解决办法:
在JAVA_HOME,也即/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home路径下新建目录Class,并执行创建软符号链接命令(推荐使用绝对路径):
sudo ln -s $JAVA_HOME/lib/tools.jar $JAVA_HOME/Classes/classes.jar
六、开始编译
进入下载的2.6.0源码根目录下,执行以下命令:
mvn package -Pdist,native -DskipTests -Dtar
等待之后会出现以下编译结果:
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.206 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [06:57 min]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [03:22 min]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.272 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [02:02 min]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [02:46 min]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [19:36 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [07:47 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [01:21 min]
[INFO] Apache Hadoop Common ............................... SUCCESS [12:20 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 4.994 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 30.076 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.044 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [07:09 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [01:46 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [03:25 min]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 3.440 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.027 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.031 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [01:08 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [01:32 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.027 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [01:21 min]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [01:49 min]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 2.149 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 4.729 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 15.844 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 4.096 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 5.619 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.024 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 1.940 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 1.541 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.031 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 3.909 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 3.328 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.040 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 17.288 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 13.444 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 2.979 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 7.477 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 5.705 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [01:24 min]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 1.265 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.280 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 3.765 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 16.991 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 31.676 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.504 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 4.793 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 3.410 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.948 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.599 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.169 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 6.456 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 4.146 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [08:32 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 6.769 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.112 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.510 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.581 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.021 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 26.965 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:28 h
[INFO] Finished at: 2016-06-28T01:40:35+08:00
[INFO] Final Memory: 193M/874M
[INFO] ------------------------------------------------------------------------
七、将编译出的native library复制到下载的二进制版本的Hadoop 2.6.0相应目录中
- 编译出的native library库的位置为
hadoop-2.6.0-src/hadoop-dist/target/hadoop-2.6.0/lib/native
- 拷贝到二进制版本的Hadoop 2.6.0的目录
hadoop-2.6.0/lib/native
八、修改/etc/hadoop/hadoop-env.sh配置
export HADOOP_OPTS=”$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.library.path=/hadoop-2.6.0/lib/native”
九、重新启动Hadoop
此时就不会出现本文开头处的那个警告了。
十、附录
ssh localhost 报错:connection closed by ::1
解决办法:
查看mac os x 系统日志/var/log/system.log
后发现有提示:
(1)Jun 28 09:29:29 bj-m-203544a.local sshd[48173]: error: Could not load host key: /etc/ssh_host_rsa_key
(2)Jun 28 09:29:29 bj-m-203544a.local sshd[48173]: error: Could not load host key: /etc/ssh_host_dsa_key
根据提示把.ssh/id_dsa
和id_rsa
拷贝到/etc目录下,并重命名为ssh_host_dsa_key
和ssh_host_rsa_key
即可。ssh localhost 免密登录
修改.ssh/authorized_keys
文件权限为644
(1)chmod 644 authorized_keys
(2)修改/etc/ssh_config文
件中属性PasswordAuthentication
值为no【可能会影响其他需要密码的ssh登录,若有影响再把该值给为yes即可。】单机模式安装的Hadoop配置打包,详见个人资源
其它
(1)启动Hadoop:
./start-dfs.sh
./start-yarn.sh
(2)关闭Hadoop
./stop-dfs.sh
./stop-yarn.sh
(3)hadoop fs -mkdir /tmp
(4)hadoop fs -copyFromLocal ~/word.txt /tmp
(5)Hadoop管理页面
http://localhost:50070
(6)yarn管理界面
http://localhost:8098/cluster
【端口实在yarn-site.xml中设置的】
(7)hadoop fs -rm /tmp/out/part-r-00000
(8)hadoop fs -rmdir /tmp/out/
备注:如果有Hadoop进程没有起来,可以查看Hadoop日志,里面会有详细的记录:
hadoop/logs/
-rw-r--r-- 1 junwei8 staff 0 Jun 28 10:16 SecurityAuth-junwei8.audit
-rw-r--r-- 1 junwei8 staff 65411 Jun 28 11:39 hadoop-junwei8-datanode-bj-m-203544a.local.log
-rw-r--r-- 1 junwei8 staff 511 Jun 28 11:39 hadoop-junwei8-datanode-bj-m-203544a.local.out
-rw-r--r-- 1 junwei8 staff 511 Jun 28 10:16 hadoop-junwei8-datanode-bj-m-203544a.local.out.1
-rw-r--r-- 1 junwei8 staff 87710 Jun 28 11:40 hadoop-junwei8-namenode-bj-m-203544a.local.log
-rw-r--r-- 1 junwei8 staff 511 Jun 28 11:39 hadoop-junwei8-namenode-bj-m-203544a.local.out
-rw-r--r-- 1 junwei8 staff 511 Jun 28 10:16 hadoop-junwei8-namenode-bj-m-203544a.local.out.1
-rw-r--r-- 1 junwei8 staff 64683 Jun 28 11:40 hadoop-junwei8-secondarynamenode-bj-m-203544a.local.log
-rw-r--r-- 1 junwei8 staff 511 Jun 28 11:39 hadoop-junwei8-secondarynamenode-bj-m-203544a.local.out
-rw-r--r-- 1 junwei8 staff 511 Jun 28 10:16 hadoop-junwei8-secondarynamenode-bj-m-203544a.local.out.1
drwxr-xr-x 2 junwei8 staff 68 Jun 28 11:40 userlogs
-rw-r--r-- 1 junwei8 staff 77547 Jun 28 11:40 yarn-junwei8-nodemanager-bj-m-203544a.local.log
-rw-r--r-- 1 junwei8 staff 494 Jun 28 11:40 yarn-junwei8-nodemanager-bj-m-203544a.local.out
-rw-r--r-- 1 junwei8 staff 494 Jun 28 10:17 yarn-junwei8-nodemanager-bj-m-203544a.local.out.1
-rw-r--r-- 1 junwei8 staff 251199 Jun 28 11:40 yarn-junwei8-resourcemanager-bj-m-203544a.local.log
-rw-r--r-- 1 junwei8 staff 494 Jun 28 11:40 yarn-junwei8-resourcemanager-bj-m-203544a.local.out
-rw-r--r-- 1 junwei8 staff 494 Jun 28 10:17 yarn-junwei8-resourcemanager-bj-m-203544a.local.out.1
-rw-r--r-- 1 junwei8 staff 494 Jun 27 21:18 yarn-junwei8-resourcemanager-bj-m-203544a.local.out.2
-rw-r--r-- 1 junwei8 staff 494 Jun 27 21:17 yarn-junwei8-resourcemanager-bj-m-203544a.local.out.3
十一、参考链接:
- .html
- =1
- .6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html
更多推荐
mac下hadoop 2.6.0编译native library
发布评论