问题描述
限时送ChatGPT账号..我有一个包含 n 列的数据框,我想用空值替换所有这些列中的空字符串.
I have a data frame with n number of columns and I want to replace empty strings in all these columns with nulls.
我尝试使用
val ReadDf = rawDF.na.replace("columnA", Map( "" -> null));
和
val ReadDf = rawDF.withColumn("columnA", if($"columnA"=="") lit(null) else $"columnA" );
它们都不起作用.
任何线索将不胜感激.谢谢.
Any leads would be highly appreciated. Thanks.
推荐答案
您的第一种方法由于一个错误而失败,该错误阻止了 replace
能够用空值替换值,请参阅 这里.
Your first approach seams to fail due to a bug that prevents replace
from being able to replace values with nulls, see here.
您的第二种方法失败了,因为您将驱动程序端 Scala 代码与执行程序端 Dataframe 指令混淆:您的 if-else 表达式将在 驱动程序 上被评估一次(而不是每条记录);你想用对 when
函数的调用来替换它;此外,要比较列的值,您需要使用 ===
运算符,而不是 Scala 的 ==
,它只是比较驱动程序端的 Column
> 对象:
Your second approach fails because you're confusing driver-side Scala code for executor-side Dataframe instructions: your if-else expression would be evaluated once on the driver (and not per record); You'd want to replace it with a call to when
function; Moreover, to compare a column's value you need to use the ===
operator, and not Scala's ==
which just compares the driver-side Column
object:
import org.apache.spark.sql.functions._
rawDF.withColumn("columnA", when($"columnA" === "", lit(null)).otherwise($"columnA"))
这篇关于用 Spark Dataframe 中的空值替换空值的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
更多推荐
[db:关键词]
发布评论