将数据从 Google Cloud Storage 导出到 Amazon S3

本文介绍了将数据从 Google Cloud Storage 导出到 Amazon S3的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我想将 BigQuery 中的一个表中的数据传输到 Redshift 中的另一个表中.我计划的数据流如下:

I would like to transfer data from a table in BigQuery, into another one in Redshift. My planned data flow is as follows:

BigQuery -> Google Cloud Storage -> Amazon S3 -> Redshift

BigQuery -> Google Cloud Storage -> Amazon S3 -> Redshift

我知道 Google Cloud Storage Transfer Service，但我不确定它是否能帮到我.来自 Google Cloud 文档:

I know about Google Cloud Storage Transfer Service, but I'm not sure it can help me. From Google Cloud documentation:

云存储传输服务

此页面描述了您可以使用的云存储传输服务将在线数据快速导入 Google Cloud Storage.

This page describes Cloud Storage Transfer Service, which you can use to quickly import online data into Google Cloud Storage.

我了解此服务可用于将数据导入 Google Cloud Storage，而不是从中导出.

I understand that this service can be used to import data into Google Cloud Storage and not to export from it.

是否可以将数据从 Google Cloud Storage 导出到 Amazon S3?

Is there a way I can export data from Google Cloud Storage to Amazon S3?

推荐答案

您可以使用 gsutil 将数据从 Google Cloud Storage 存储桶复制到 Amazon 存储桶，使用的命令如下:

You can use gsutil to copy data from a Google Cloud Storage bucket to an Amazon bucket, using a command such as:

gsutil -m rsync -rd gs://your-gcs-bucket s3://your-s3-bucket

请注意，上面的 -d 选项会导致 gsutil rsync 从您的 S3 存储桶中删除 GCS 存储桶中不存在的对象(除了添加新对象).如果您只想将新对象从 GCS 添加到 S3 存储桶，则可以不使用该选项.

Note that the -d option above will cause gsutil rsync to delete objects from your S3 bucket that aren't present in your GCS bucket (in addition to adding new objects). You can leave off that option if you just want to add new objects from your GCS to your S3 bucket.