使用PySpark将JSON文件读取为Pyspark Dataframe吗?

本文介绍了使用PySpark将JSON文件读取为Pyspark Dataframe吗?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

如何读取以下JSON结构以使用PySpark触发数据帧?

How can I read the following JSON structure to spark dataframe using PySpark?

我的JSON结构

{"results":[{"a":1,"b":2,"c":"name"},{"a":2,"b":5,"c":"foo"}]}

我尝试过:

df = spark.read.json('simple.json');

我希望将输出a，b，c作为列，并将值作为相应的行.

I want the output a,b,c as columns and values as respective rows.

谢谢.

推荐答案

Json字符串变量

如果您将 json字符串作为变量，则可以

simple_json = '{"results":[{"a":1,"b":2,"c":"name"},{"a":2,"b":5,"c":"foo"}]}' rddjson = sc.parallelize([simple_json]) df = sqlContext.read.json(rddjson) from pyspark.sql import functions as F df.select(F.explode(df.results).alias('results')).select('results.*').show(truncate=False)

这将为您提供

+---+---+----+ |a |b |c | +---+---+----+ |1 |2 |name| |2 |5 |foo | +---+---+----+

Json字符串作为文件(sparkContext和sqlContext)中的单独行

如果文件中有 json字符串作为单独的行 ，则可以如上所述使用sparkContext将其读取到rdd [string] 中，其余过程相同如上

If you have json strings as separate lines in a file then you can read it using sparkContext into rdd[string] as above and the rest of the process is same as above

rddjson = sc.textFile('/home/anahcolus/IdeaProjects/pythonSpark/test.csv') df = sqlContext.read.json(rddjson) df.select(F.explode(df['results']).alias('results')).select('results.*').show(truncate=False)

Json字符串作为文件中的单独行(仅适用于sqlContext)

如果文件中有 json字符串作为单独的行 ，则只能使用sqlContext.但是该过程很复杂，因为您必须为其创建架构

If you have json strings as separate lines in a file then you can just use sqlContext only. But the process is complex as you have to create schema for it

df = sqlContext.read.text('path to the file') from pyspark.sql import functions as F from pyspark.sql import types as T df = df.select(F.from_json(df.value, T.StructType([T.StructField('results', T.ArrayType(T.StructType([T.StructField('a', T.IntegerType()), T.StructField('b', T.IntegerType()), T.StructField('c', T.StringType())])))])).alias('results')) df.select(F.explode(df['results.results']).alias('results')).select('results.*').show(truncate=False)

应该与上述结果相同

我希望答案会有所帮助