本文介绍了Spark DataFrame按键将列值聚合到列表中的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有一个看起来像这样的DataFrame:
I have a DataFrame that looks like this:
+-----------------+-------+ |Id | value | +-----------------+-------+ | 1622| 139685| | 1622| 182118| | 1622| 127955| | 3837|3224815| | 1622| 727761| | 1622| 155875| | 3837|1504923| | 1622| 139684| +-----------------+-------+我想把它变成:
+-----------------+-------------------------------------------+ |Id | value | +-----------------+-------------------------------------------+ | 1622|139685,182118,127955,727761,155875,139684 | | 3837|3224815,1504923 | +-----------------+-------------------------------------------+仅使用DataFrame函数可能吗?还是需要将其转换为和RDD?
Is this possible with DataFrame functions only or do I need to convert it to and RDD?
推荐答案使用DataFrame API可能.试试:
It is possible with the DataFrame API. Try:
df.groupBy(col("Id")) .agg(collect_list(col("value")) as "value")如果您要用,分隔的String而不是Array,请尝试以下操作:
If instead of an Array you want a String separated by ,, then try this:
df.groupBy(col("Id")) .agg(collect_list(col("value")) as "value") .withColumn("value", concat_ws(",", col("value")))更多推荐
Spark DataFrame按键将列值聚合到列表中
发布评论