如何将df列的每个元素转换为不同的列？

本文介绍了如何将df列的每个元素转换为不同的列？的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

请假设我在Pyspark上有这样的数据框；

please assume that I have a data frame on Pyspark like this;

import pandas from pyspark.sql import SparkSession from pyspark.sql.functions import explode spark = SparkSession \ .Builder() \ .appName('stackoverflow') \ .getOrCreate() data = { 'location_id': [1, 2, 3], 'product_model_features': [ [{'key': 'A', 'value': 'B'}, {'key': 'C', 'value': 'D'}, {'key': 'E', 'value': 'F'}], [{'key': 'A', 'value': 'H'}, {'key': 'E', 'value': 'J'}], [{'key': 'C', 'value': 'N'}, {'key': 'E', 'value': 'P'}] ] } df = pandas.DataFrame(data) df = spark.createDataFrame(df) df = df.withColumn('p', explode('product_model_features')) \ .select('location_id', 'p.key', 'p.value') df.show()

输出为

+-----------+---+-----+ |location_id|key|value| +-----------+---+-----+ | 1| A| B| | 1| C| D| | 1| E| F| | 2| A| H| | 2| E| J| | 3| C| N| | 3| E| P| +-----------+---+-----+

我想将键列值转换为带有值的其他列。在下面，您可以看到输出内容。如果您对pyspark有任何想法，请告诉我

I want to convert column "key" values to a different column with values. Below you can see what output looks like. please let me know if you have idea on pyspark

+-----------+----+----+-+ |location_id|A |C |E| +-----------+----+----+-+ | 1|B |D |F| | 2|H |Null|J| | 3|Null|N |P| +-----------+----+----+-+

推荐答案

您正在寻找 pivot（）函数来转换数据框。

You're looking for pivot() function to transform your dataframe.

import pandas from pyspark.sql import SparkSession from pyspark.sql.functions import explode, col, first spark = SparkSession \ .Builder() \ .appName('stackoverflow') \ .getOrCreate() data = { 'location_id': [1, 2, 3], 'product_model_features': [ [{'key': 'A', 'value': 'B'}, {'key': 'C', 'value': 'D'}, {'key': 'E', 'value': 'F'}], [{'key': 'A', 'value': 'H'}, {'key': 'E', 'value': 'J'}], [{'key': 'C', 'value': 'N'}, {'key': 'E', 'value': 'P'}] ] } df = pandas.DataFrame(data) df = spark.createDataFrame(df) df = df \ .withColumn('p', explode('product_model_features')) \ .select('location_id', 'p.key', 'p.value') df = df \ .groupby('location_id') \ .pivot('key') \ .agg(first('value')) \ .sort('location_id') df.show()

输出：

+-----------+----+----+---+ |location_id| A| C| E| +-----------+----+----+---+ | 1| B| D| F| | 2| H|null| J| | 3|null| N| P| +-----------+----+----+---+