本文介绍了行模式作为 PySpark DataFrame 中的新列的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
限时送ChatGPT账号..是否可以根据前一列的最大值添加新列,其中前一列是字符串文字.考虑以下数据框:
Is it possible to add a new column based on the maximum of previous columns where the previous columns are string literals. Consider following dataframe:
df = spark.createDataFrame(
[
('1',25000,"black","black","white"),
('2',16000,"red","black","white"),
],
['ID','cash','colour_body','colour_head','colour_foot']
)
那么目标框架应该是这样的:
Then the target frame should look like this:
df = spark.createDataFrame(
[
('1',25000,"black","black","white", "black" ),
('2',16000,"red","black","white", "white" ),
],
['ID','cash','colour_body','colour_head','colour_foot', 'max_v']
)
如果没有可检测的最大值,则应使用最后一个有效颜色.
If there is no maximum detectable, then the last valid colour should be used.
是否有某种计数器可用或 udf 的可能性?
Is there some kind of counter possibility available or udf?
推荐答案
定义一个围绕 statistics.mode
的 UDF 来计算具有所需语义的行模式:
Define a UDF around statistics.mode
to compute the row-wise mode with the required semantics:
import statistics
from pyspark.sql.functions import udf, col
from pyspark.sql.types import StringType
def mode(*x):
try:
return statistics.mode(x)
except statistics.StatisticsError:
return x[-1]
mode = udf(mode, StringType())
df.withColumn("max_v", mode(*[col(c) for c in df.columns if 'colour' in c])).show()
+---+-----+-----------+-----------+-----------+-----+
| ID| cash|colour_body|colour_head|colour_foot|max_v|
+---+-----+-----------+-----------+-----------+-----+
| 1|25000| black| black| white|black|
| 2|16000| red| black| white|white|
+---+-----+-----------+-----------+-----------+-----+
这篇关于行模式作为 PySpark DataFrame 中的新列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
更多推荐
[db:关键词]
发布评论