使用火花发射器时将参数传递给罐子(Pass parameters to the jar when using spark launcher)

使用火花发射器时将参数传递给罐子(Pass parameters to the jar when using spark launcher)

我正在尝试创建一个可执行的jar，它使用spark启动器来运行另一个包含数据转换任务的jar（该jar创建spark会话）。

我需要将java参数（一些java数组）传递给由启动器执行的jar。

object launcher { @throws[Exception] // How do I pass parameters to spark_job_with_spark_session.jar def main(args: Array[String]): Unit = { val handle = new SparkLauncher() .setAppResource("spark_job_with_spark_session.jar") .setVerbose(true) .setMaster("local[*]") .setConf(SparkLauncher.DRIVER_MEMORY, "4g") .launch() } }

我怎样才能做到这一点？

I am trying to create an executable jar which is using a spark launcher to run another jar with data transformation task(this jar creates spark session).

I need to pass java parameters(some java arrays) to the jar which is executed by the launcher.

object launcher { @throws[Exception] // How do I pass parameters to spark_job_with_spark_session.jar def main(args: Array[String]): Unit = { val handle = new SparkLauncher() .setAppResource("spark_job_with_spark_session.jar") .setVerbose(true) .setMaster("local[*]") .setConf(SparkLauncher.DRIVER_MEMORY, "4g") .launch() } }

How can I do that?

最满意答案

需要传递java参数（一些java数组）

它相当于执行spark-submit所以你不能直接传递Java对象。 使用应用程序参数

addAppArgs(String... args)

传递应用程序参数，并在应用程序中解析它们。

need to pass java parameters(some java arrays)

It is equivalent to executing spark-submit so you cannot pass Java objects directly. Use app args

addAppArgs(String... args)

to pass application arguments, and parse them in your app.