Google Cloud Composer和Google Cloud SQL

本文介绍了Google Cloud Composer和Google Cloud SQL的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我们可以通过哪些方式从新推出的Google Cloud Composer连接到Google Cloud SQL（MySQL）实例？目的是将数据从Cloud SQL实例获取到BigQuery（可能通过Cloud Storage进行中间步骤）。

Can

如果不能，则可以通过Kubernetes引入Cloud SQL代理吗？服务经纪人？ -> cloud.google/kubernetes -engine / docs / concepts / add-on / service-broker

应该使用Airflow计划和调用GCP API命令，例如1）将mysql表导出到云存储2）将mysql导出到bigquery吗？

也许我缺少其他方法来完成此操作

解决方案

Cloud SQL代理提供了对Cloud SQL Second Generation实例的安全访问，而无需将IP地址列入白名单或配置SSL。 - Google CloudSQL代理文档

CloudSQL代理似乎是连接到CloudSQL的首选方法。因此，在Composer中，从1.6.1版开始，我们可以创建一个新的Kubernetes Pod来运行gcr.io/cloudsql-docker/gce-proxy:latest映像，通过服务公开它，然后在Composer中创建一个Connection以使用

要进行设置：

• 关注 Google的文档

• 使用来自阿里克的中号帖子

  • 检查是否已创建广告连播 kubectl get pods --all-namespaces

  • 检查是否已创建服务 kubectl get services --all-namespaces

  • 跳转到工作节点 kubectl --namespace = composer-1-6-1-airflow -1-10-1-< some-uid> exec -it airflow-worker-< some-uid> bash

    • 测试mysql连接 mysql -u composer -p --host< service-name> .default.svc.cluster.local

注意：

• Composer现在使用命名空间来组织广告连播

• 位于不同名称空间的豆荚不要互相交谈除非您为其提供完整路径< k8-service-name>。< k8-namespace-name> .svc.cluster.local

• 使用以下命令创建新的 Composer连接完整路径将启用成功连接

What ways do we have available to connect to a Google Cloud SQL (MySQL) instance from the newly introduced Google Cloud Composer? The intention is to get data from a Cloud SQL instance into BigQuery (perhaps with an intermediary step through Cloud Storage).

Can the Cloud SQL proxy be exposed in some way on pods part the Kubernetes cluster hosting Composer?

If not can the Cloud SQL Proxy be brought in by using the Kubernetes Service Broker? -> cloud.google/kubernetes-engine/docs/concepts/add-on/service-broker

Should Airflow be used to schedule and call GCP API commands like 1) export mysql table to cloud storage 2) read mysql export into bigquery?

Perhaps there are other methods that I am missing to get this done

解决方案

"The Cloud SQL Proxy provides secure access to your Cloud SQL Second Generation instances without having to whitelist IP addresses or configure SSL." -Google CloudSQL-Proxy Docs

CloudSQL Proxy seems to be the recommended way to connect to CloudSQL above all others. So in Composer, as of release 1.6.1, we can create a new Kubernetes Pod to run the gcr.io/cloudsql-docker/gce-proxy:latest image, expose it through a service, then create a Connection in Composer to use in the operator.

To get set up:

• Follow Google's documentation

• Test the connection using info from Arik's Medium Post

  • Check that the pod was created kubectl get pods --all-namespaces

  • Check that the service was created kubectl get services --all-namespaces

  • Jump into a worker node kubectl --namespace=composer-1-6-1-airflow-1-10-1-<some-uid> exec -it airflow-worker-<some-uid> bash

    • Test mysql connection mysql -u composer -p --host <service-name>.default.svc.cluster.local

Notes:

• Composer now uses namespaces to organize pods

• Pods in different namespaces don't talk to each other unless you give them the full path <k8-service-name>.<k8-namespace-name>.svc.cluster.local

• Creating a new Composer Connection with the full path will enable successful connection