Azkaban配置SparkJobType

Azkaban配置SparkJobType"/>

Azkaban配置SparkJobType

1.首先要安装azkaban的插件：
azkaban-jobtype：

2.将spark、common.properties、commonprivate.properties拷贝到服务器中对应的目录，最终的文件展示如下

[hadoop@bigdata01 jobtypes]$ tree
.
├── commonprivate.properties
├── common.properties
└── spark├── plugin.properties└── private.properties

3.配置commonprivate.properties中hadoop.home和spark.home指定的家目录

4.配置common.properties中hadoop.home和spark.home指定的家目录

5.将common.properties和commonprivate.properties 中azkaban.should.proxy=false设置为false

6.编辑spark目录下的两个文件

plugin.properties

queue=default

private.properties

jobtype.class=azkaban.jobtype.HadoopSparkJob将原来的替换成一下这种形式jobtype.classpath=hadoop.classpath:{spark.home}/conf:${spark.home}/lib/*

7.conf/azkaban.properties增加一个配置

端口号可以随便写
azkaban.webserver.url=https://bigdata01:8666

8.编写配置文件内容：wordcount.flow

config:user.to.proxy: hadoop
nodes:- name: sparkwctype: sparkconfig:class: com.ruozedata.wordcount.WordCountApp2master: yarndeploy-mode: clientexecutor-memory: 512Mdriver-memory: 512Mconf.spark.testing.memory: 471859200execution-jar: spark-core-1.0.jarjars: spark-utils-1.0.jarparams: hdfs://bigdata01:9000/data/word.txt hdfs://bigdata01:9000/spark/job/out

9.将execution-jar和jars 对应的jar包一起打包到zip中提交到web界面运行