使用火花发射器时将参数传递给罐子(Pass parameters to the jar when using spark launcher)
我正在尝试创建一个可执行的jar,它使用spark启动器来运行另一个包含数据转换任务的jar(该jar创建spark会话)。
我需要将java参数(一些java数组)传递给由启动器执行的jar。
object launcher { @throws[Exception] // How do I pass parameters to spark_job_with_spark_session.jar def main(args: Array[String]): Unit = { val handle = new SparkLauncher() .setAppResource("spark_job_with_spark_session.jar") .setVerbose(true) .setMaster("local[*]") .setConf(SparkLauncher.DRIVER_MEMORY, "4g") .launch() } }我怎样才能做到这一点?
I am trying to create an executable jar which is using a spark launcher to run another jar with data transformation task(this jar creates spark session).
I need to pass java parameters(some java arrays) to the jar which is executed by the launcher.
object launcher { @throws[Exception] // How do I pass parameters to spark_job_with_spark_session.jar def main(args: Array[String]): Unit = { val handle = new SparkLauncher() .setAppResource("spark_job_with_spark_session.jar") .setVerbose(true) .setMaster("local[*]") .setConf(SparkLauncher.DRIVER_MEMORY, "4g") .launch() } }How can I do that?
最满意答案
需要传递java参数(一些java数组)
它相当于执行spark-submit所以你不能直接传递Java对象。 使用应用程序参数
addAppArgs(String... args)传递应用程序参数,并在应用程序中解析它们。
need to pass java parameters(some java arrays)
It is equivalent to executing spark-submit so you cannot pass Java objects directly. Use app args
addAppArgs(String... args)to pass application arguments, and parse them in your app.
更多推荐
发布评论