Google Cloud Composer 和 Google Cloud SQL

Cloud SQL 代理能否以某种方式在托管 Composer 的 Kubernetes 集群的 Pod 上公开?

Can the Cloud SQL proxy be exposed in some way on pods part the Kubernetes cluster hosting Composer?

如果不能，是否可以使用 Kubernetes Service Broker 引入 Cloud SQL Proxy?-> cloud.google/kubernetes-engine/docs/concepts/add-on/service-broker

If not can the Cloud SQL Proxy be brought in by using the Kubernetes Service Broker? -> cloud.google/kubernetes-engine/docs/concepts/add-on/service-broker

是否应该使用 Airflow 来调度和调用 GCP API 命令，例如 1) 将 mysql 表导出到云存储 2) 将 mysql 导出读入 bigquery?

Should Airflow be used to schedule and call GCP API commands like 1) export mysql table to cloud storage 2) read mysql export into bigquery?

也许我还缺少其他方法来完成这项工作

Perhaps there are other methods that I am missing to get this done

Cloud SQL 代理提供对 Cloud SQL 第二代实例的安全访问，而无需将 IP 地址列入白名单或配置 SSL."-Google CloudSQL-Proxy 文档

"The Cloud SQL Proxy provides secure access to your Cloud SQL Second Generation instances without having to whitelist IP addresses or configure SSL." -Google CloudSQL-Proxy Docs

CloudSQL 代理似乎是最推荐的连接 CloudSQL 的方式.所以在 Composer 中，从 1.6.1 版本开始，我们可以创建一个新的 Kubernetes Pod 来运行 gcr.io/cloudsql-docker/gce-proxy:latest 镜像，通过一个服务公开它，然后在 Composer 中创建一个 Connection 来使用在运算符中.

CloudSQL Proxy seems to be the recommended way to connect to CloudSQL above all others. So in Composer, as of release 1.6.1, we can create a new Kubernetes Pod to run the gcr.io/cloudsql-docker/gce-proxy:latest image, expose it through a service, then create a Connection in Composer to use in the operator.

开始设置:

• 遵循 Google 的文档

使用来自 Arik 的中篇帖子

• 检查 pod 是否已创建 kubectl get pods --all-namespaces

检查服务是否已创建kubectl get services --all-namespaces

跳转到一个工作节点 kubectl --namespace=composer-1-6-1-airflow-1-10-1-exec -itairflow-worker-<some-uid>bash

• 测试mysql连接mysql -u composer -p --host .default.svc.cluster.local

注意事项:

• Composer 现在使用命名空间来组织豆荚

不同命名空间中的 Pod 不相互交谈 除非你给他们完整路径 ..svc.cluster.local

Pods in different namespaces don't talk to each other unless you give them the full path <k8-service-name>.<k8-namespace-name>.svc.cluster.local

使用完整路径创建新的 Composer Connection 将启用成功连接

Creating a new Composer Connection with the full path will enable successful connection