如何在Data Science Experience上实施TensorFrames Spark软件包?

本文介绍了如何在Data Science Experience上实施TensorFrames Spark软件包?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我已经能够导入软件包:

I've been able to import the package:

import pixiedust pixiedust.installPackage("databricks:tensorframes:0")

但是当我尝试一个简单的例子时:

But when I try a simple example:

import tensorflow as tf import tensorframes as tfs from pyspark.sql import Row data = [Row(x=[float(x), float(2 * x)], key=str(x % 2), z = float(x+1)) for x in range(1, 6)] df = spark.createDataFrame(data) tfs.print_schema(df)

我收到以下错误:

...

Py4JJavaError: An error occurred while calling o97.loadClass. : java.lang.NoClassDefFoundError:com.typesafe.scalalogging.slf4j.LazyLogging

...

我已经查看了这个问题，似乎有依赖项树中的较​​旧的scala-logging-slf4j工件. 如何删除此工件?假定删除后，我可以使用PixieDust添加新版本:

I've looked up the issue and it seems like there is an older scala-logging-slf4j artifact in the dependency tree. How do I remove this artifact? Once it's removed, I assume I can add a newer version with PixieDust:

pixiedust.installPackage("mvnrepository/artifact/com.typesafe.scala-logging/scala-logging-slf4j_2.10/2.1.2") pixiedust.installPackage("mvnrepository/artifact/com.typesafe.scala-logging/scala-logging-api_2.10/2.1.2")

pixiedust.installPackage("mvnrepository/artifact/com.typesafe.scala-logging/scala-logging-slf4j_2.10/2.1.2") pixiedust.installPackage("mvnrepository/artifact/com.typesafe.scala-logging/scala-logging-api_2.10/2.1.2")

推荐答案

IBM支持小组的查尔斯帮助我找到了要包括的罐子.

Charles from IBM support helped me find which jars to include:

pixiedust.installPackage("central.maven/maven2/com/typesafe/scala-logging/scala-logging-slf4j_2.10/2.1.2/scala-logging-slf4j_2.10-2.1.2.jar") pixiedust.installPackage("central.maven/maven2/com/typesafe/scala-logging/scala-logging-api_2.10/2.1.2/scala-logging-api_2.10-2.1.2.jar")

pixiedust.installPackage("central.maven/maven2/com/typesafe/scala-logging/scala-logging-slf4j_2.10/2.1.2/scala-logging-slf4j_2.10-2.1.2.jar") pixiedust.installPackage("central.maven/maven2/com/typesafe/scala-logging/scala-logging-api_2.10/2.1.2/scala-logging-api_2.10-2.1.2.jar")

这从技术上解决了第一个错误，但是TesnsorFrames仍然无法正常工作.我将发布另一个更具体的问题.

This technically solves the first error, however TesnsorFrames still aren't working. I'll post another question that is more specific.