我有一个具有以下结构的JSON流,这些结构已转换为数据框
I've a stream of JSONs with following structure that gets converted to dataframe
{ "a": 3936, "b": 123, "c": "34", "attributes": { "d": "146", "e": "12", "f": "23" } }数据框显示功能导致以下输出
The dataframe show functions results in following output
sqlContext.read.json(jsonRDD).show +----+-----------+---+---+ | a| attributes| b| c| +----+-----------+---+---+ |3936|[146,12,23]|123| 34| +----+-----------+---+---+如何将属性列(嵌套的JSON结构)分为 attributes.d,attributes.e和attributes.f ,作为 seperate 列到一个新的数据框中,所以可以在新数据框中具有a,b,c,attributes.d,attributes.e和attributes.f列?
How can I split attributes column (nested JSON structure) into attributes.d, attributes.e and attributes.f as seperate columns into a new dataframe, so I can have columns as a, b, c, attributes.d, attributes.e and attributes.f in the new dataframe?
推荐答案
-
如果要将列从a命名为f:
df.select("a", "b", "c", "attributes.d", "attributes.e", "attributes.f") -
如果要使用以attributes.前缀命名的列:
If you want columns named with attributes. prefix:
df.select($"a", $"b", $"c", $"attributes.d" as "attributes.d", $"attributes.e" as "attributes.e", $"attributes.f" as "attributes.f")-
如果您的列名是从外部来源(例如配置)提供的:
If names of your columns are supplied from an external source (e.g. configuration):
val colNames: Seq("a", "b", "c", "attributes.d", "attributes.e", "attributes.f") df.select(colNames.head, colNames.tail: _*).toDF(colNames:_*)
更多推荐
Spark数据帧将嵌套的JSON转换为单独的列
发布评论