Spark Twitter Streaming 异常:(org.apache.spark.Logging) classnotfound

本文介绍了Spark Twitter Streaming 异常:(org.apache.spark.Logging) classnotfound的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在使用 Maven 在 Scala 中尝试 Spark Twitter Streaming 示例，但在运行它时出现以下错误:

<块引用>

引起:java.lang.ClassNotFoundException:org.apache.spark.Logging

以下是我的依赖项:

<依赖><groupId>org.apache.spark</groupId><artifactId>spark-core_2.10</artifactId><version>2.0.0</version></依赖><依赖><groupId>org.apache.spark</groupId><artifactId>spark-streaming_2.10</artifactId><version>2.0.0</version></依赖><依赖><groupId>org.apache.spark</groupId><artifactId>spark-streaming-twitter_2.11</artifactId><version>1.6.2</version></依赖>

我知道 Logging 已经移到 org.apache.spark.internal.Logging 但我不知道是不是这个原因，我已经试过了将依赖项的版本更改为最新版本，但没有运气.

解决方案

TLDR;

Class org.apache.spark.Logging 在 Spark 版本 1.5.2 或更低(虽然我没有在所有较低版本上测试)但在高于相同的版本中不可用.<小时>这一切都归结为使用不兼容版本的Apache Spark:

1.让我们尝试在 import org.apache.spark.Logging代码>Spark 2.0.0:

<块引用>

user@ubuntu:~$/opt/spark/bin/spark-shell欢迎来到____ __/__/__ ___ _____//___ /_ /_ `/__/'_//___/.__/\_,_/_//_/\_ 2.0.0 版/_/使用 Scala 版本 2.11.8(Java HotSpot(TM) 64 位服务器 VM，Java 1.8.0_101)标度>导入 org.apache.spark.Logging<console>:23: 错误:对象日志记录不是包 org.apache.spark 的成员导入 org.apache.spark.Logging^

未找到类 org.apache.spark.Logging.

<小时>

2.让我们尝试在 <上import org.apache.spark.Logging代码>Spark 1.6.2:

(与上述相同，即未找到类 org.apache.spark.Logging.)

---

3.让我们尝试在 <import org.apache.spark.Logging代码>Spark 1.5.2:

<块引用>

user@ubuntu:~$/opt/spark-1.5.2-bin-hadoop2.6/bin/spark-shell欢迎来到____ __/__/__ ___ _____//___ /_ /_ `/__/'_//___/.__/\_,_/_//_/\_ 版本 1.5.2/_/使用 Scala 版本 2.10.4(Java HotSpot(TM) 64 位服务器 VM，Java 1.8.0_101)标度>导入 org.apache.spark.Logging导入 org.apache.spark.Logging

是的！可用并成功导入

如您所见，org.apache.spark.Loggingstreaming-twitter_2.11/1.6.2">Spark-Streaming-Twitter，在 Spark 版本中可用 1.5.2 或更低，所以我建议你使用 1.5.2 或更低版本的 spark.

因此，您应该使用以下内容替换您的 Maven 依赖项:(假设您使用的是 Scala 2.11.x)

<依赖><groupId>org.apache.spark</groupId><artifactId>spark-core_2.11</artifactId><version>1.5.2</version></依赖><依赖><groupId>org.apache.spark</groupId><artifactId>spark-streaming_2.11</artifactId><version>1.5.2</version></依赖><依赖><groupId>org.apache.spark</groupId><artifactId>spark-streaming-twitter_2.11</artifactId><version>1.6.2</version></依赖>

注意 artifactId: 2.11 指的是 scala 版本和 version: 1.5.2 或 1.6.2 指的是库(spark-core 或 spark-streaming-twitter)版本.

I am trying Spark Twitter Streaming example with Scala using Maven but I am getting below error when I run it:

Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging

Below are my dependencies:

<dependencies>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>2.0.0</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.10</artifactId>
    <version>2.0.0</version>
</dependency> 
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-twitter_2.11</artifactId>
    <version>1.6.2</version> 
</dependency> 

I know that the Logging has been moved to org.apache.spark.internal.Logging but I don't know if it is the reason , I already tried to change the version of dependencies to the latest one but with no luck.

解决方案

TLDR;

Class org.apache.spark.Logging is available in Spark version 1.5.2 or lower (though I didn't test on all lower versions) but is not available in versions higher than the same.

---

It all comes down to using incompatible version of Apache Spark:

1. Let's try to import org.apache.spark.Logging on Spark 2.0.0:

user@ubuntu:~$ /opt/spark/bin/spark-shell
Welcome to
  ____              __
 / __/__  ___ _____/ /__
_ / _ / _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_   version 2.0.0
   /_/      
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
scala> import org.apache.spark.Logging
<console>:23: error: object Logging is not a member of package org.apache.spark
import org.apache.spark.Logging
          ^

Class org.apache.spark.Logging is not found.

---

2. Let's try to import org.apache.spark.Logging on Spark 1.6.2:

(same as above i.e. Class org.apache.spark.Logging is not found.)

---

3. Let's try to import org.apache.spark.Logging on Spark 1.5.2:

user@ubuntu:~$ /opt/spark-1.5.2-bin-hadoop2.6/bin/spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_   version 1.5.2
      /_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
scala> import org.apache.spark.Logging
import org.apache.spark.Logging

YES! It is available and successfully imported

As you can see that org.apache.spark.Logging which is required by the Spark-Streaming-Twitter, is available in Spark version 1.5.2 or lower, so I would recommend you to use 1.5.2 or a lower version of spark.

Hence, you should replace your maven dependencies with followings: (Assuming that you are using Scala 2.11.x)

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>1.5.2</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>1.5.2</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-twitter_2.11</artifactId>
    <version>1.6.2</version>
</dependency>

Note that the artifactId: 2.11 refers to scala version and version: 1.5.2 or 1.6.2 refers to the library (spark-core or spark-streaming-twitter) version.

这篇关于Spark Twitter Streaming 异常:(org.apache.spark.Logging) classnotfound的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！