在PySpark中爆炸

本文介绍了在PySpark中爆炸的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我想从包含单词列表的DataFrame转换为每个单词都在其自己行中的DataFrame.

I would like to transform from a DataFrame that contains lists of words into a DataFrame with each word in its own row.

如何在DataFrame的列上爆炸?

How do I do explode on a column in a DataFrame?

这是我的一些尝试的示例，您可以在其中取消注释每个代码行并获取以下注释中列出的错误.我在带有Spark 1.6.1的Python 2.7中使用PySpark.

Here is an example with some of my attempts where you can uncomment each code line and get the error listed in the following comment. I use PySpark in Python 2.7 with Spark 1.6.1.

from pyspark.sql.functions import split, explode DF = sqlContext.createDataFrame([('cat \n\n elephant rat \n rat cat', )], ['word']) print 'Dataset:' DF.show() print '\n\n Trying to do explode: \n' DFsplit_explode = ( DF .select(split(DF['word'], ' ')) # .select(explode(DF['word'])) # AnalysisException: u"cannot resolve 'explode(word)' due to data type mismatch: input to function explode should be array or map type, not StringType;" # .map(explode) # AttributeError: 'PipelinedRDD' object has no attribute 'show' # .explode() # AttributeError: 'DataFrame' object has no attribute 'explode' ).show() # Trying without split print '\n\n Only explode: \n' DFsplit_explode = ( DF .select(explode(DF['word'])) # AnalysisException: u"cannot resolve 'explode(word)' due to data type mismatch: input to function explode should be array or map type, not StringType;" ).show()

请咨询

推荐答案

explode和split是SQL函数.两者都在SQL Column上运行. split将Java正则表达式作为第二个参数.如果要在任意空格上分离数据，则需要这样的东西:

explode and split are SQL functions. Both operate on SQL Column. split takes a Java regular expression as a second argument. If you want to separate data on arbitrary whitespace you'll need something like this:

df = sqlContext.createDataFrame( [('cat \n\n elephant rat \n rat cat', )], ['word'] ) df.select(explode(split(col("word"), "\s+")).alias("word")).show() ## +--------+ ## | word| ## +--------+ ## | cat| ## |elephant| ## | rat| ## | rat| ## | cat| ## +--------+