Azkaban 简介及安装(3.70)

简介及安装(3.70)"/>

Azkaban 简介及安装(3.70)

Azkaban 简介及安装(3.70)

1.Azkaban 简介

官网：/

Azkaban 一个批量工作流任务调度器，使用Java语言开发。用于在一个工作流内以一个特定的顺序运行一组工作和流程。Azkaban使用job配置文件建立任务之间的依赖关系，并提供一个易于使用的web用户界面维护和跟踪你的工作流。

Azkaban 的优点

• 提供功能清晰，简单易用的Web UI界面

• 提供job配置文件快速建立任务和任务之间的依赖关系

• 提供模块化和可插拔的插件机制，原生支持command、Java、Hive、Pig、Hadoop

• 基于Java开发，代码结构清晰，易于二次开发

Azkaban 安装模式

Azkaban 有三种部署方式：单服务模式、2个服务模式、分布式多服务模式

• solo server model（单服务模式）：该模式中 webServer 和 executorServer 运行在同一个进程中，进程名是AzkabanSingleServer。可以使用自带的H2数据库或者配置mysql数据。该模式适用于小规模的使用。此模式合适个人试用使用。它也可以用在小规模的使用案例中。

• two server model（2个服务模式）：数据库为mysql，采用主从设置进行备份，管理服务器（webServer）和执行服务器（executorServer）在不同进程中运行，这种模式下，管理服务器和执行服务器互不影响。适用在较重的生成环境中。

• multiple-executor（分布式多服务模式）：存放元数据的数据库为mysql，采用主从设置进行备份，管理服务器（webServer）和执行服务器（executorServer）在不同进程中运行。多个 Executor 模式为最重的生产环境。

2.Azkaban 3.x 安装(two-server 模式)

安装前准备

• 安装JDK1.8
• 安装MySQL

下载源码

git clone .git

编译源码

cd azkaban; ./gradlew build installDist

编译报错：

> Task :az-crypto:test azkaban.crypto.DecryptionTest > testV1_1 FAILEDjava.lang.RuntimeException: java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machineat azkaban.crypto.Crypto.decrypt(Crypto.java:76)at azkaban.crypto.DecryptionTest.testV1_1(DecryptionTest.java:35)Caused by:java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machineat azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:57)at azkaban.crypto.Crypto.decrypt(Crypto.java:74)... 1 moreCaused by:org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machineat org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:1050)at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:725)at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:55)... 2 moreazkaban.crypto.EncryptionTest > testEncryption FAILEDorg.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machineat org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.encrypt(StandardPBEByteEncryptor.java:924)at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.encrypt(StandardPBEStringEncryptor.java:642)at azkaban.crypto.CryptoV1_1.encrypt(CryptoV1_1.java:42)at azkaban.crypto.Crypto.encrypt(Crypto.java:58)at azkaban.crypto.EncryptionTest.testEncryption(EncryptionTest.java:28)5 tests completed, 2 failed

解决方法：

1.下载jre：.html

2.在/usr/java/jdk1.8.0_141-cloudera/jre/lib/security目录下替换原有的

3.重新编译 ./gradlew build installDist

启停单机版的服务

cd azkaban-solo-server/build/install/azkaban-solo-server
bin/start-solo.sh 
bin/shutdown-solo.sh

数据库设置

# 建用户 建库 授权 刷新权限
CREATE DATABASE azkaban_zxl;
CREATE USER 'azkaban_zxl'@'%' IDENTIFIED BY 'azkaban_zxl';
GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban_zxl.* to 'azkaban_zxl'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;# 在/etc/myf中配置Mysql Packet Size的大小
[mysqld]
...
max_allowed_packet=1024M# 重启mysql
/sbin/service mysqld restart# 创建Azkaban表
source /root/apps/create-all-sql.sql

配置web-server

vi /root/azkaban/azkaban-web-server/conf/azkaban.properties default.timezone.id=Asia/Shanghai
#database.type=h2
#h2.path=./h2
#h2.create.tables=true
database.type=mysql
mysql.port=3306
mysql.host=10.xxx.xxx.xx
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
executor.port=12321
azkaban.use.multiple.executors=true将azkaban-web-server/conf/azkaban.properties 同步到 /root/azkaban/azkaban-exec-server/conf下

启动分布式多服务模式

多服务模式是多个executorServer分布在 不同服务器上，只需要将/root/azkaban/azkaban-exec-server拷贝到不同机器上即可组成分布式。

先启动executorServer bin/start-exec.sh, 再启动webServer bin/start-web.sh。

**Note:**启动executorServer时会将executorServer的host,port同步到mysql的executors表中;

root@localhost|azkaban_zxl>select * from executors;
+----+---------------------------------------+-------+--------+
| id | host                                  | port  | active |
+----+---------------------------------------+-------+--------+
|  1 | 10.xxx.xx.xx 					     | 12321 |      0 |
|  2 | 10.xxx.xx.xx 						 | 12321 |      0 |
+----+---------------------------------------+-------+--------+

启动webServer时会如下错误，是因为自动同步到executors表中的executor默认是未激活，需要在数据库中update：

2019/03/22 10:00:47.210 +0800 INFO [ExecutorManager] [Azkaban] Initializing executors from database.
2019/03/22 10:00:47.418 +0800 ERROR [ExecutorManager] [Azkaban] No active executors found
2019/03/22 10:00:47.418 +0800 ERROR [StdOutErrRedirect] [Azkaban] Exception in thread "main" 
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban] azkaban.executor.ExecutorManagerException: No active executors found
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.executor.ActiveExecutors.setupExecutors(ActiveExecutors.java:52)
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.executor.ExecutorManager.setupExecutors(ExecutorManager.java:231)
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.executor.ExecutorManager.initialize(ExecutorManager.java:155)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.executor.ExecutorManager.start(ExecutorManager.java:169)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.webapp.AzkabanWebServer.launch(AzkabanWebServer.java:235)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:228)

root@localhost|azkaban_zxl>update executors set active=1 where id=1;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0root@localhost|azkaban_zxl>update executors set active=1 where id=2;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

executor重启方式

1. 停止webServer    bin/shutdown-web.sh 
2. 停止executor  bin/shutdown-exec.sh 
3. 删除 xxx/azkaban-exec-server-3.65.0/currentpid 文件
4. 启动executor  bin/start-exec.sh
5. 去数据库中确认 executor状态为1
6. 启动webServer    bin/start-web.sh 

shylin