pyspark写入csv文件到S3错误(pyspark writing csv file to S3 error)

pyspark写入csv文件到S3错误(pyspark writing csv file to S3 error)

我正在使用pyspark，并且在写入S3时遇到麻烦，但从S3读取并不是问题。

这是我的代码：

dic = {'a': {'c1(%)': 0.0, 'c2': 0, 'c3($)': 260, 'c4(%)': 4.79, 'c5': 78, 'c6': 352}, 'b': {'c1(%)': 0.0, 'c2': 0, 'c3($)': 5, 'c4(%)': 0.09, 'c5': 2, 'c6': 280}, 'c': {'c1(%)': 0.0, 'c2': 0, 'c3($)': 0, 'c4(%)': 0.0, 'c5': 0, 'c6': 267}} df = pd.DataFrame(dic) df.to_csv("s3://work/.../filename_2018-01-04_08:50:45.csv")

这是错误：

IOError: [Errno 2] No such file or directory: 's3://work/.../filename_2018-01-04_08:50:45.csv'

问题是什么？

I am using pyspark and I am having trouble writing to S3, but reading from S3 is not a problem.

this is my code:

dic = {'a': {'c1(%)': 0.0, 'c2': 0, 'c3($)': 260, 'c4(%)': 4.79, 'c5': 78, 'c6': 352}, 'b': {'c1(%)': 0.0, 'c2': 0, 'c3($)': 5, 'c4(%)': 0.09, 'c5': 2, 'c6': 280}, 'c': {'c1(%)': 0.0, 'c2': 0, 'c3($)': 0, 'c4(%)': 0.0, 'c5': 0, 'c6': 267}} df = pd.DataFrame(dic) df.to_csv("s3://work/.../filename_2018-01-04_08:50:45.csv")

this is the error:

IOError: [Errno 2] No such file or directory: 's3://work/.../filename_2018-01-04_08:50:45.csv'

what is the problem?

最满意答案

看到我上面的评论，你需要使用Spark DataFrame。 一个简单的方法可以将Pandas DF的索引转换为列，然后转换为火花DF：

df2=sqlContext.createDataFrame(df.reset_index(drop=False))

然后使用：

df2.write.save("s3://work/.../filename_2018-01-04_08:50:45.csv", format='csv', header=True)

See my comment above, you need to use a Spark DataFrame. One easy way to accomplish this would be to turn the index on the Pandas DF into a column and then convert to spark DF:

df2=sqlContext.createDataFrame(df.reset_index(drop=False))

Then use:

df2.write.save("s3://work/.../filename_2018-01-04_08:50:45.csv", format='csv', header=True)