CDH5.16.1集群企业真正离线部署

离线部署"/>

CDH5.16.1集群企业真正离线部署

CDH5.16.1集群离线部署

离线部署节点规划：

节点	MySQL组件	Parcel文件离线源	CM服务进程	大数据组件
Hadoop001	MySQL	Parcel	Activity Monitor	NN RM DN NM
Hadoop002			Alert Publisher Event Server	DN NM
Hadoop003			Host Monitor Service Monitor	DN NM

离线部署步骤：

• mysql离线部署
• CM 离线部署
• Parcel离线部署

下载源：

JDK
.html
下载jdk-8u202-linux-x64.tar.gz

MySQL
.7.html#downloads

MySQL JDBC Jar
.1.47/mysql-connector-java-5.1.47.jar
使用的时候去掉版本号mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar

CM
.16.1_x86_64.tar.gz

Parcel
.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
.16.1/manifest.json

集群机器的准备工作：

1.所有节点配置Linux机器的hosts文件

192.168.52.200 hadoop001
192.168.52.210 hadoop002
192.168.52.230 hadoop003

2.所有节点禁用防火墙并清空规则

查看防火墙状态：firewall-cmd --state
关闭防火墙：systemctl stop firewalld
禁用防火墙：systemctl disable firewalld
清空规则：iptables -F

3.关闭所有节点的selinux

vi /etc/selinux/config

将SELINUX=enforcing改为SELINUX=disabled  重启生效

4.设置所有节点时钟同步

查看时间及时区

[root@hadoop001 ~]# timectl
-bash: timectl: command not found
[root@hadoop001 ~]# timedatectlLocal time: Wed 2018-01-29 17:38:16 CSTUniversal time: Wed 2018-01-29 09:38:16 UTCRTC time: Wed 2018-01-29 09:38:17Time zone: Asia/Shanghai (CST, +0800)NTP enabled: no
NTP synchronized: yesRTC in local TZ: noDST active: n/a

查看timedatectl 使用帮助

[root@hadoop001 ~]# timedatectl --help
timedatectl [OPTIONS...] COMMAND ...Query or change system time and date settings.-h --help                Show this help message--version             Show package version--no-pager            Do not pipe output into a pager--no-ask-password     Do not prompt for password-H --host=[USER@]HOST    Operate on remote host-M --machine=CONTAINER   Operate on local container--adjust-system-clock Adjust system clock when changing local RTC modeCommands:status                   Show current time settingsset-time TIME            Set system timeset-timezone ZONE        Set system time zonelist-timezones           Show known time zonesset-local-rtc BOOL       Control whether RTC is in local timeset-ntp BOOL             Control whether NTP is enabled

配置时钟同步：这里以hadoop001 联网进行时钟同步，其他节点与hadoop001同步
安装ntp yum install -y ntp

hadoop001 配置：
vi /etc/ntp.conf 增加如下内容

#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#当外部时间不可⽤时，可使⽤本地硬件时间
server 127.127.1.0 iburst local clock
#允许哪些⽹段的机器来同步时间
restrict 192.168.52.0 mask 255.255.255.0 nomodify notrap

开启ntp及查看状态：

[root@hadoop001 ~]#  systemctl start ntpd
[root@hadoop001 ~]# systemctl status ntpd
● ntpd.service - Network Time ServiceLoaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)Active: active (running) since Wed 2018-01-29 17:11:32 CST; 32min agoProcess: 9170 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)Main PID: 9200 (ntpd)CGroup: /system.slice/ntpd.service└─9200 /usr/sbin/ntpd -u ntp:ntp -gJan 29 17:11:34 hadoop001 ntpd_intres[9202]: DNS 2.asia.pool.ntp.org -> 120.25.115.20
Jan 29 17:11:34 hadoop001 ntpd_intres[9202]: DNS 3.asia.pool.ntp.org -> 150.95.187.128
Jan 29 17:11:34 hadoop001 ntpd_intres[9202]: DNS 0.centos.pool.ntp.org -> 84.16.67.12
Jan 29 17:11:34 hadoop001 ntpd_intres[9202]: DNS 1.centos.pool.ntp.org -> 78.46.102.180
Jan 29 17:11:34 hadoop001 ntpd_intres[9202]: DNS 2.centos.pool.ntp.org -> 95.216.200.137
Jan 29 17:11:34 hadoop001 ntpd_intres[9202]: DNS 3.centos.pool.ntp.org -> 193.182.111.14
Jan 29 17:11:34 hadoop001 ntpd[9200]: Listen normally on 4 ens33 192.168.52.200 UDP 123
Jan 29 17:11:34 hadoop001 ntpd[9200]: new interface(s) found: waking up resolver
Jan 29 17:11:36 hadoop001 ntpd[9200]: Listen normally on 5 ens33 fe80::d8d:fbba:1681:5d8 UDP 123
Jan 29 17:11:36 hadoop001 ntpd[9200]: new interface(s) found: waking up resolver

其他从节点停⽌禁⽤ntpd服务，并向hadoop001 进行时钟同步
从节点执行：

[root@hadoop002 ~]# systemctl stop ntpd
[root@hadoop002 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop002 ~]# /usr/sbin/ntpdate hadoop001

设置定时任务每天隔10分钟同步hadoop001节点时间[root@hadoop002 ~]# crontab -e
*/10 * * * * /usr/sbin/ntpdate hadoop001

安装JDK
将jdk统一安装到/usr/java 下面
mkdir /usr/java
tar -xzvf jdk-8u45-linux-x64.tar.gz -C /usr/java/
#切记必须修正所属⽤户及⽤户组
chown -R root:root /usr/java/jdk1.8.0_45
配置环境变量：
vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATHsource /etc/profile
which java

hadoop001节点 安装MySQL
创建CDH的元数据库和⽤户、amon服务的数据库及⽤户

create database cmf DEFAULT CHARACTER SET utf8;
create database amon DEFAULT CHARACTER SET utf8;
grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY 'bigdata123456!';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'bigdata123456!';
flush privileges;

hadoop001 部署mysql驱动包

mkdir -p /usr/share/java/
cp mysql-connector-java.jar /usr/share/java/

CDH部署

1.离线部署cm server及agent

1.1.所有节点创建⽬录及解压
mkdir /opt/cloudera-manager
tar -zxvf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager/

1.2.所有节点修改agent的配置，指向server的节点hadoop001
vi /opt/cloudera-manager/cm5.16.1/etc/cloudera-scm-agent/config.ini

# Hostname of the CM server.
server_host=hadoop001

1.3.主节点修改server的配置:
vi /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop001
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=Ruozedata123456!
com.cloudera.cmf.db.setupType=EXTERNAL

1.4.所有节点创建⽤户

useradd --system --home=/opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

1.5.⽬录修改⽤户及⽤户组
chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

2.hadoop001节点部署离线parcel源

2.1.部署离线parcel源

$ mkdir -p /opt/cloudera/parcel-repo
$ cp CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel /opt/cloudera/parcel-repo/

#切记cp时，重命名去掉1，不然在部署过程CM认为如上⽂件下载未完整，会持续下载

$ cp CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha
$ cp manifest.json /opt/cloudera/parcel-repo/

2.2.⽬录修改⽤户及⽤户组

$ chown -R cloudera-scm:cloudera-scm /opt/cloudera/

3.所有节点创建软件安装⽬录、⽤户及⽤户组权限

mkdir -p /opt/cloudera/parcels chown -R cloudera-scm:cloudera-scm /opt/cloudera/

4.hadoop001节点启动Server

4.1.启动server
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start

4.2.云主机web界⾯，设置该hadoop001节点防⽕墙放开7180端⼝

4.3.等待1min，打开 http://hadoop001:7180 账号密码:admin/admin

4.4.假如打不开，去看server的log，根据错误仔细排查错误

4.5使用tail -F /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-server/cloudera-scm-server.log监控日志

5.所有节点启动Agent

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

6.接下来，全部Web界⾯操作