如何使用Apache Livy设置Spark配置属性?

本文介绍了如何使用Apache Livy设置Spark配置属性?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

在将Spark作业提交给Apache Livy时，我不知道如何以编程方式传递SparkSession参数:

I don't know how to pass SparkSession parameters programmatically when submitting Spark job to Apache Livy:

这是Test Spark作业:

This is the Test Spark job:

class Test extends Job[Int]{ override def call(jc: JobContext): Int = { val spark = jc.sparkSession() // ... } }

这是将Spark作业提交给Livy的方式:

This is how this Spark job is submitted to Livy:

val client = new LivyClientBuilder() .setURI(new URI(livyUrl)) .build() try { client.uploadJar(new File(testJarPath)).get() client.submit(new Test()) } finally { client.stop(true) }

如何将以下配置参数传递给SparkSession?

How can I pass the following configuration parameters to SparkSession?

.config("es.nodes","1localhost") .config("es.port",9200) .config("es.nodes.wan.only","true") .config("es.index.auto.create","true")

推荐答案

您可以像这样通过LivyClientBuilder轻松地做到这一点:

You can do that easily through the LivyClientBuilder like this:

val client = new LivyClientBuilder() .setURI(new URI(livyUrl)) .setConf("es.nodes","1localhost") .setConf("key", "value") .build()