问题描述
限时送ChatGPT账号..我有一个 Pandas 数据框.我试图首先将包含字符串值的两列连接到一个列表中,然后使用 zip,我用_"连接了列表的每个元素.我的数据集如下:
I have a Pandas dataframe. I have tried to join two columns containing string values into a list first and then using zip, I joined each element of the list with '_'. My data set is like below:
df['column_1']: 'abc, def, ghi'
df['column_2']: '1.0, 2.0, 3.0'
我想将这两列连接到第三列中,如下所示,用于数据框的每一行.
I wanted to join these two columns in a third column like below for each row of my dataframe.
df['column_3']: [abc_1.0, def_2.0, ghi_3.0]
我已经使用下面的代码在 python 中成功地做到了这一点,但数据帧非常大,需要很长时间才能为整个数据帧运行它.我想在 PySpark 中做同样的事情以提高效率.我已经成功读取了 spark 数据帧中的数据,但是我很难确定如何使用 PySpark 等效函数复制 Pandas 函数.如何在 PySpark 中获得我想要的结果?
I have successfully done so in python using the code below but the dataframe is quite large and it takes a very long time to run it for the whole dataframe. I want to do the same thing in PySpark for efficiency. I have read the data in spark dataframe successfully but I'm having a hard time determining how to replicate Pandas functions with PySpark equivalent functions. How can I get my desired result in PySpark?
df['column_3'] = df['column_2']
for index, row in df.iterrows():
while index < 3:
if isinstance(row['column_1'], str):
row['column_1'] = list(row['column_1'].split(','))
row['column_2'] = list(row['column_2'].split(','))
row['column_3'] = ['_'.join(map(str, i)) for i in zip(list(row['column_1']), list(row['column_2']))]
我已使用以下代码将两列转换为 PySpark 中的数组
I have converted the two columns to arrays in PySpark by using the below code
from pyspark.sql.types import ArrayType, IntegerType, StringType
from pyspark.sql.functions import col, split
crash.withColumn("column_1",
split(col("column_1"), ",\s*").cast(ArrayType(StringType())).alias("column_1")
)
crash.withColumn("column_2",
split(col("column_2"), ",\s*").cast(ArrayType(StringType())).alias("column_2")
)
现在我需要的是使用_"压缩两列中数组的每个元素.我该如何使用 zip 呢?任何帮助表示赞赏.
Now all I need is to zip each element of the arrays in the two columns using '_'. How can I use zip with this? Any help is appreciated.
推荐答案
与 Python 等效的 Spark SQL 将是 pyspark.sql.functions.arrays_zip
:
A Spark SQL equivalent of Python's would be pyspark.sql.functions.arrays_zip
:
pyspark.sql.functions.arrays_zip(*cols)
集合函数:返回一个合并的结构数组,其中第 N 个结构包含输入数组的所有第 N 个值.
Collection function: Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.
所以如果你已经有两个数组:
So if you already have two arrays:
from pyspark.sql.functions import split
df = (spark
.createDataFrame([('abc, def, ghi', '1.0, 2.0, 3.0')])
.toDF("column_1", "column_2")
.withColumn("column_1", split("column_1", "\s*,\s*"))
.withColumn("column_2", split("column_2", "\s*,\s*")))
您可以将其应用于结果
from pyspark.sql.functions import arrays_zip
df_zipped = df.withColumn(
"zipped", arrays_zip("column_1", "column_2")
)
df_zipped.select("zipped").show(truncate=False)
+------------------------------------+
|zipped |
+------------------------------------+
|[[abc, 1.0], [def, 2.0], [ghi, 3.0]]|
+------------------------------------+
现在组合结果,您可以transform
(如何使用变换高阶函数?a>, TypeError: Column is not iterable - How to iterate over ArrayType()?):
Now to combine the results you can transform
(How to use transform higher-order function?, TypeError: Column is not iterable - How to iterate over ArrayType()?):
df_zipped_concat = df_zipped.withColumn(
"zipped_concat",
expr("transform(zipped, x -> concat_ws('_', x.column_1, x.column_2))")
)
df_zipped_concat.select("zipped_concat").show(truncate=False)
+---------------------------+
|zipped_concat |
+---------------------------+
|[abc_1.0, def_2.0, ghi_3.0]|
+---------------------------+
注意:
高阶函数 transform
和 arrays_zip
已在 Apache Spark 2.4 中引入.
Higher order functions transform
and arrays_zip
has been introduced in Apache Spark 2.4.
这篇关于如何在 Spark SQL 中压缩两个数组列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
更多推荐
[db:关键词]
发布评论