如何在ScalaTest测试中正确使用Spark?

本文介绍了如何在ScalaTest测试中正确使用Spark?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我有多个ScalaTest类，它们使用 BeforeAndAfterAll 构造一个 SparkContext 并随后将其停止，如下所示:

class MyTest使用BeforeAndAfterAll {私人var sc:SparkContext = null覆盖受保护的def beforeAll():单位= {sc = ...//创建SparkContext}覆盖保护的def afterAll():单位= {sc.stop()}//我的测试如下}

这些测试从IntelliJ IDEA启动时运行良好，但是运行 sbt测试时，我得到 WARN SparkContext:正在构造另一个SparkContext(或在其构造函数中引发了异常).这可能表示一个错误，因为在此JVM中只能运行一个SparkContext(请参阅SPARK-2243).，然后，我认为还有许多其他与此问题相关的异常.>

如何正确使用Spark?我是否必须为整个测试套件创建一个全局 SparkContext ，如果是，该怎么做?

解决方案

似乎我看不见树木的树木，我在 build.sbt 中忘记了以下内容:测试中的

parallelExecution:=否

在此行中，测试运行.

I have multiple ScalaTest classes which use BeforeAndAfterAll to construct a SparkContext and stop it afterwards like so:

class MyTest extends FlatSpec with Matchers with BeforeAndAfterAll { private var sc: SparkContext = null override protected def beforeAll(): Unit = { sc = ... // Create SparkContext } override protected def afterAll(): Unit = { sc.stop() } // my tests follow }

These tests run fine when started from IntelliJ IDEA, but when running sbt test, I get WARN SparkContext: Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext may be running in this JVM (see SPARK-2243)., and after that, a bunch of other exceptions which are, I suppose, related to this issue.

How to correctly use Spark? Do I have to create one global SparkContext for the whole test suite, and if yes, how do I do this?

解决方案

Seems like I lost the sight of the wood for the trees, I forgot the following line in my build.sbt:

parallelExecution in test := false

With this line, the test runs.