本文介绍了从AWS S3 pyspark读取文件数据的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我在s3中放置了一个json文件. s3网址类似于以下网址:
I have a json file placed in s3. The s3 url is similar to the below one:
s3-eu-region-1.amazonaws/dir-resources/sample.json但是在pyspark中传递相同的内容时,它没有读取文件.
But in pyspark when pass the same, It is not reading the file.
path = "s3-eu-region-1.amazonaws/dir-resources/sample.json" df=spark.read.json(path)但是我可以通过浏览器下载它.
But I am able to download it through browser.
推荐答案假定dir-resources是存储桶的名称,您应该能够使用以下URI访问文件:
Assuming that dir-resources is the name of your bucket, you should be able to access to the file with the following URI:
path = "s3://dir-resources/sample.json"
在某些情况下,您可能不得不使用s3n协议:
In some cases, you may have to use the s3n protocol instead:
path = "s3n://dir-resources/sample.json"
更多推荐
从AWS S3 pyspark读取文件数据
发布评论