在 Datastax Spark 提交中使用 Scala 从 S3 存储桶读取文件到 Spark 数据帧，给出 AWS 错误消息:错误请求

编程入门行业动态更新时间:2024-10-25 18:25:42

本文介绍了在 Datastax Spark 提交中使用 Scala 从 S3 存储桶读取文件到 Spark 数据帧，给出 AWS 错误消息:错误请求的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

限时送ChatGPT账号..

我正在尝试读取位于孟买地区的 s3 存储桶上的 CSV 文件.我正在尝试使用 datastax dse spark-submit 读取文件.

I'm trying to read CSV files which are on s3 bucket which is located in Mumbai Region.I'm trying to read the files using datastax dse spark-submit.

我尝试将 hadoop-aws 版本更改为各种其他版本.目前，hadoop-aws 版本为 2.7.3

I tried changing hadoop-aws version to various other versions. Currently, hadoop-aws version is 2.7.3

spark.sparkContext.hadoopConfiguration.set("com.amazonaws.services.s3.enableV4", "true")

spark.sparkContext.hadoopConfiguration.set("fs.s3a.endpoint", "s3.ap-south-1.amazonaws")

spark.sparkContext.hadoopConfiguration.set("fs.s3a.access.key", accessKeyId)

spark.sparkContext.hadoopConfiguration.set("fs.s3a.secret.key", secretAccessKey)

spark.sparkContext.hadoopConfiguration.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")

val df = spark.read.csv("s3a://bucket_path/csv_name.csv")

执行后，以下是我得到的错误，

Upon Executing, Following is the error which I'm getting,

线程main"中的异常com.amazonaws.services.s3.model.AmazonS3Exception:状态代码:400，AWS 服务:Amazon S3，AWS 请求 ID:8C7D34A38E359FCE，AWS 错误代码:null，AWS 错误消息:错误请求在com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)在com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)在com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)在com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)在com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)在com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)在org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)在org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)在 org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) 在org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)在 org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)在 org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) 在org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) 在org.apache.spark.sql.execution.datasources.DataSource$$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:616)在org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350)在org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350)在scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)在scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)在 scala.collection.immutable.List.foreach(List.scala:392) 在scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)在 scala.collection.immutable.List.flatMap(List.scala:355) 在org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:349)在org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)在org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:533)在org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:412)

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 8C7D34A38E359FCE, AWS Error Code: null, AWS Error Message: Bad Request at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.spark.sql.execution.datasources.DataSource$$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:616) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.immutable.List.flatMap(List.scala:355) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:349) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178) at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:533) at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:412)