单机部署"/>
大数据学习之一——Hadoop单机部署
1 背景
由于我本人服务器有限,因此尝试进行单机安装,主要为了学习使用。
2 前提
hadoop安装依赖JDK。
我这里尝试用下JDK18看看。
下载地址:/
这里咱们选择Linux/x64的版本。
2.1 配置JDK环境变量
# vi /etc/profile
在文件最后补充:
export JAVA_HOME=/usr/local/java/jdk-18.0.2.1
export PATH=$PATH:$JAVA_HOME/bin
配置生效
# source /etc/profile
4 下载
.3.4/
在这个链接,你可以看到各个版本,我这里选择的是3.3.4
在服务器上下载命令:
# wget .3.4/hadoop-3.3.4.tar.gz
5 解压
我这边上传到centos上,进行解压
# tar -zxvf hadoop-3.3.4.tar.gz
6 配置环境变量
同样,在/etc/profile文件中配置hadoop配置文件:
# vi /etc/profile在文件末尾补充:
export HADOOP_HOME=/usr/local/hadoop/hadoop-3.3.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin配置生效
# source /etc/profile
检查配置是否生效
[root@instance-1mfp2mc1 jdk-18.0.2.1]# hadoop version
Hadoop 3.3.4
Source code repository .git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb
Compiled by stevel on 2022-07-29T12:32Z
Compiled with protoc 3.7.1
From source with checksum fb9dd8918a7b8a5b430d61af858f6ec
This command was run using /usr/local/hadoop/hadoop-3.3.4/share/hadoop/common/hadoop-common-3.3.4.jar
7 配置Hadoop文件
所有配置文件都在etc目录下:hadoop-3.3.4/etc/hadoop,所以,我们只要关心这个目录下相关文件的配置。
各个配置文件说明如下:
配置文件的名称 | 作用 |
---|---|
hadoop-env.sh | 主要配置我们的java路径 |
core-site.xml | 核心配置文件,主要定义了我们文件访问的格式 hdfs:// |
hdfs-site.xml | 主要定义配置我们的hdfs的相关配置 |
mapred-site.xml | 主要定义我们的mapreduce相关的一些配置 |
slaves | 控制我们的从节点在哪里 datanode nodemanager在哪些机器上 |
yarm-site.xml | 配置我们的resourcemanager资源调度 |
7.1 修改hadoop-env.sh
在文件末尾加上如下配置:
export JAVA_HOME=/usr/local/java/jdk-18.0.2.1
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
7.2 修改 core-site.xml
将配置文件修改为:
<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9820</value></property><property><name>hadoop.tmp.dir</name><value>/usr/hadoop/hadoopdata</value></property>
</configuration>
7.3 修改hdfs-site.xml
将配置文件修改为如下,表示只有一个副本,因为我这里是单机:
<configuration><property><name>dfs.replication</name><value>1</value></property>
</configuration>
7.4 修改mapred-site.xml
将配置文件修改为:
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property>
</configuration>
7.5 修改yarn-site.xml
将配置文件修改为:
<configuration><!-- Site specific YARN configuration properties --><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value></property>
</configuration>
8 启动
8.1 对安装目录进行授权
# chmod -R 777 hadoop-3.3.4
8.2 格式化hdfs文件系统
# cd /usr/local/hadoop/hadoop-3.3.4/
# bin/hdfs namenode -format
8.3 启动hadoop
[root@instance-1mfp2mc1 hadoop-3.3.4]# sbin/start-all.sh
Starting namenodes on [localhost]
Last login: Wed Sep 14 21:45:50 CST 2022 from 218.76.52.120 on pts/3
Last failed login: Wed Sep 14 22:25:26 CST 2022 from 158.69.32.51 on ssh:notty
There were 10 failed login attempts since the last successful login.
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting datanodes
Last login: Wed Sep 14 22:28:54 CST 2022 on pts/3
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting secondary namenodes [instance-1mfp2mc1]
Last login: Wed Sep 14 22:28:54 CST 2022 on pts/3
instance-1mfp2mc1: Warning: Permanently added 'instance-1mfp2mc1,192.168.16.4' (ECDSA) to the list of known hosts.
instance-1mfp2mc1: root@instance-1mfp2mc1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting resourcemanager
Last login: Wed Sep 14 22:28:55 CST 2022 on pts/3
Starting nodemanagers
Last login: Wed Sep 14 22:28:59 CST 2022 on pts/3
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
不过,我这里失败了。
9 配置ssh并再次启动
操作步骤我就直接贴出来了:
[root@instance-1mfp2mc1 .ssh]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:5UT8ePM907UkTWhPTy5b8lExBD1Jbfs3woLFZaNwOY0 root@instance-1mfp2mc1
The key's randomart image is:
+---[RSA 3072]----+
| .. +.*=+|
| o.E B **|
| *o* *+=|
| +.++.o*=|
| S +..ooB*|
| . . oo=*|
| . . =|
| |
| |
+----[SHA256]-----+
[root@instance-1mfp2mc1 .ssh]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@instance-1mfp2mc1 .ssh]# ssh localhost
Activate the web console with: systemctl enable --now cockpit.socketLast failed login: Wed Sep 14 22:32:14 CST 2022 from 158.69.32.51 on ssh:notty
There were 4 failed login attempts since the last successful login.
Last login: Wed Sep 14 22:29:01 2022
再次启动下hadoop:
[root@instance-1mfp2mc1 ~]# cd /usr/local/hadoop/hadoop-3.3.4/
[root@instance-1mfp2mc1 hadoop-3.3.4]# sbin/start-all.sh
Starting namenodes on [localhost]
Last login: Wed Sep 14 22:32:53 CST 2022 from ::1 on pts/4
Last failed login: Wed Sep 14 22:34:01 CST 2022 from 91.240.118.222 on ssh:notty
There was 1 failed login attempt since the last successful login.
Starting datanodes
Last login: Wed Sep 14 22:35:12 CST 2022 on pts/4
Starting secondary namenodes [instance-1mfp2mc1]
Last login: Wed Sep 14 22:35:15 CST 2022 on pts/4
Starting resourcemanager
Last login: Wed Sep 14 22:35:20 CST 2022 on pts/4
Starting nodemanagers
Last login: Wed Sep 14 22:35:26 CST 2022 on pts/4
[root@instance-1mfp2mc1 hadoop-3.3.4]#
打开namenode,http://120.48.16.193:9870/,namenode的作用是存储元数据。
打开datanode,http://120.48.16.193:9864/,datanode作用是存储真正的数据。
更多推荐
大数据学习之一——Hadoop单机部署
发布评论