4040和8080上的Web UI有什么区别?

编程入门行业动态更新时间:2024-10-27 06:21:37

本文介绍了4040和8080上的Web UI有什么区别?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！问题描述

有两种不同的Web UI(一种仅用于独立模式).当我以独立模式启动Spark时，可以在端口4040上使用Web UI吗? (例如:spark-class.cmd org.apache.spark.deploy.master.Master-web ui 8080在运行，4040-在不行.)这些UI之间的主要区别是什么?

There are two different web UIs (one is for standalone mode only). Can I use web UI on port 4040 when I am launching Spark in standalone mode? (example:spark-class.cmd org.apache.spark.deploy.master.Master- web ui 8080 is working, 4040 - not.) What is the main difference between these UIs?

我是否可以启动Spark(没有hadoop，hdfs，yarn等)，保持它的状态并将我的jar(类)提交给它?我想看完工作统计.我正在尝试这样的事情:

Is it possible for me to launch Spark (without hadoop, hdfs, yarn etc), to keep it up and to submit my jars(classes) into it? I want to watch job statistics after it finishes. I am trying something like this:

Server: Spark\bin>spark-class.cmd org.apache.spark.deploy.master.Master Worker: Spark\bin>spark-class.cmd org.apache.spark.deploy.worker.Worker spark://169.254.8.45:7077 --cores 4 --memory 512M Submit: Spark\bin>spark-submit.cmd --class demo.TreesSample --master spark://169.254.8.45:7077 file:///E:/spark-demo/target/demo.jar

它运行.为此，它将在端口4040上获取新的WebUI.我在8080的师父界面中看不到任何东西.

It runs. It gets new WebUI on port 4040 up for this task. I dont see anything in Master's ui on 8080.

当前，我正在使用win7 x64，spark-1.5.2-bin-hadoop2.6.如果有关系，我可以切换到linux.

Currently I'm using win7 x64, spark-1.5.2-bin-hadoop2.6. I can switch into linux if it matters.

推荐答案

如spark.master.ui.port或SPARK_MASTER_WEBUI_PORT更改独立Master的Web UI端口. spark.apache/docs/latest/security.html#standalone-mode-only"rel =" nofollow noreferrer>仅为网络安全性/独立模式配置端口.

You should be able to change the web UI port for standalone Master using spark.master.ui.port or SPARK_MASTER_WEBUI_PORT as described in Configuring Ports for Network Security / Standalone mode only.

Standalone Master的Web UI是集群管理器的管理控制台(它恰好是Apache Spark的一部分，但可能像Hadoop YARN和Apache Mesos一样是单独的产品).话虽如此，这通常会使两个Web UI的共同之处混淆，而答案却无济于事.

Standalone Master's web UI is a management console of a cluster manager (that happens to be part of Apache Spark, but could've been a separate product as Hadoop YARN and Apache Mesos). Having said that, it can often be confusing what the two web UIs have in common, and the answer is nothing.

Spark驱动程序的Web UI用来显示您的计算进度(作业，阶段，用于RDD持久性的存储，广播，累加器)，而独立的Master的Web UI可以让您知道操作环境"的当前状态( aka Spark独立集群).

The Spark driver's web UI is to show the progress of your computations (jobs, stages, storage for RDD persistence, broadcasts, accumulators) while standalone Master's web UI is to let you know the current state of your "operating environment" (aka the Spark Standalone cluster).