在Kubernetes上运行Spark示例失败

编程入门行业动态更新时间:2024-10-27 22:23:34

本文介绍了在Kubernetes上运行Spark示例失败的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！问题描述

我的Mac OS/X版本:10.15.3
Minikube版本:1.9.2

我使用以下命令启动minikube，无需任何额外配置.

I start the minikube use the following command without any extra configuration.

minikube start --driver=virtualbox --image-repository='registry-hangzhou.aliyuncs/google_containers' --cpus 4 --memory 4096 --alsologtostderr

然后我从Spark官方网站下载spark-2.4.5-bin-hadoop2.7并通过以下命令构建spark图像

And I download spark-2.4.5-bin-hadoop2.7 from the Spark official website and build spark images by the following command

eval $(minikube docker-env) ./bin/docker-image-tool.sh -m -t 2.4.5 build

然后我在存储Spark 2.4.5的本地计算机中使用follwing命令运行Spark-pi.

then I run Spark-pi using the follwing command within my local machine where store the Spark 2.4.5.

kubectl create serviceaccount spark kubectl create clusterrolebinding spark-role --clusterrole=admin --serviceaccount=default:spark --namespace=default ./bin/spark-submit \ --master k8s://192.168.99.104:8443 \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.executor.instances=1 \ --conf spark.kubernetes.container.image=spark:2.4.5 \ local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar

我收到以下错误

完整日志可以在完整日志中找到

谁能解释这个错误以及如何解决?

Can anyone explain this error and how to solve it?

推荐答案

请检查您使用Minikube启动的Kubernetes版本.

Please check the Kubernetes version you launched with Minikube.

Spark v2.4.5 fabric8 Kubernetes客户端v4.6.1与直至1.15.2的Kubernetes API兼容(请参阅 answer ).

Spark v2.4.5 fabric8 Kubernetes client v4.6.1 is compatible with Kubernetes API up to 1.15.2 (refer answer).

您可以通过在minikube start命令中添加--kubernetes-version标志来使用Minikube启动特定的Kubernetes API版本(请参考文档).

You can launch the specific Kubernetes API version with Minikube by adding --kubernetes-version flag to minikube start command (refer docs).

此问题也可能是由此问题的注释中描述的OkHttp库错误引起的.