使用pyspark将文本文件保存到S3时遇到问题。我可以保存到S3,但它首先上传到S3上的_temporary,然后继续复制到预期的位置。这显着增加了工作时间。我试图编译一个DirectFileOutputComter,它应该直接写入想要的S3 url,但我无法让Spark使用这个类。
I am having a problem saving text files to S3 using pyspark. I am able to save to S3, but it first uploads to a _temporary on S3 and then proceeds to copy to the intended location. This increases the jobs run time significantly. I have attempted to compile a DirectFileOutputComitter which should write directly to the intended S3 url, but I cannot get Spark to utilize this class.
示例:
someRDD.saveAsTextFile(s3a:// somebucket / savefolder)这会创建一个
this creates a
s3a:// somebucket / savefolder / _temporary /s3a://somebucket/savefolder/_temporary/
$ b $ p
目录,然后写入之后,S3复制操作将文件移回到
directory which is then written to after which a S3 copy operation moves the files back to
s3a:// somebucket / savefolder
s3a://somebucket/savefolder
我的问题是,任何人都有DirectFileOutputCommiter的工作罐,或者任何人有经验这个问题。
My question is does anyone have a working jar of the DirectFileOutputCommiter, or if anyone has experience working around this issue.
相关链接:
Relevant Links:
推荐答案
我能解决这个问题通过用Databricks中的DirectOutputCommitter修补Hadoop 2.7.2并将修补过的jar部署到我的spark中nstances。下面链接是一个Git仓库补丁。
I was able to fix this issue by patching Hadoop 2.7.2 with a DirectOutputCommitter from Databricks and deploying the patched jar to my spark instances. Linked below is a git repo with the patched jar.
Github链接
更多推荐
Spark 1.6 DirectFileOutputCommitter
发布评论