我想在DataFrame中复制一行,该怎么做?
I want to duplicate a Row in a DataFrame, how can I do that?
例如,我有一个由1行组成的DataFrame,并且我想制作一个具有100个相同行的DataFrame.我想出了以下解决方案:
For example, I have a DataFrame consisting of 1 Row, and I want to make a DataFrame with 100 identical Rows. I came up with the following solution:
var data:DataFrame=singleRowDF for(i<-1 to 100-1) { data = data.unionAll(singleRowDF) }但这引入了许多转换,而且看来我随后的动作非常缓慢.还有另一种方法吗?
But this introduces many transformations and it seems my subsequent actions become very slow. Is there another way to do it?
推荐答案您可以添加一列,其字面值为Array,其大小为100,然后使用explode使其每个元素创建自己的行;然后,只需删除此虚拟"列即可:
You can add a column with a literal value of an Array with size 100, and then use explode to make each of its elements create its own row; Then, just get rid of this "dummy" column:
import org.apache.spark.sql.functions._ val result = singleRowDF .withColumn("dummy", explode(array((1 until 100).map(lit): _*))) .selectExpr(singleRowDF.columns: _*)更多推荐
N次复制Spark Row
发布评论