从AWS S3 pyspark读取文件数据

本文介绍了从AWS S3 pyspark读取文件数据的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我在s3中放置了一个json文件. s3网址类似于以下网址:

I have a json file placed in s3. The s3 url is similar to the below one:

s3-eu-region-1.amazonaws/dir-resources/sample.json

但是在pyspark中传递相同的内容时，它没有读取文件.

But in pyspark when pass the same, It is not reading the file.

path = "s3-eu-region-1.amazonaws/dir-resources/sample.json" df=spark.read.json(path)

但是我可以通过浏览器下载它.

But I am able to download it through browser.

推荐答案

假定dir-resources是存储桶的名称，您应该能够使用以下URI访问文件:

Assuming that dir-resources is the name of your bucket, you should be able to access to the file with the following URI:

path = "s3://dir-resources/sample.json"

在某些情况下，您可能不得不使用s3n协议:

In some cases, you may have to use the s3n protocol instead:

path = "s3n://dir-resources/sample.json"