损坏的DAG：（...）没有名为docker的模块

本文介绍了损坏的DAG：（...）没有名为docker的模块的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我所有的BigQuery连接器都在运行，但是我希望在Cloud Composer而不是App Engine Flexible上安排一些在Docker容器中的现有脚本。

下面的脚本似乎遵循我可以找到的示例：

import datetime 从airflow导入DAG 从airflow导入模型从airflow.operators.docker_operator导入DockerOperator 昨天= datetime.datetimebine（ datetime.datetime.today（）-datetime.timedelta（1） ， datetime.datetime.min.time（）） default_args = {＃将开始日期设置为昨天将立即启动DAG 'start_date'：昨天， ＃如果任务失败，请等待至少5分钟后重试一次。'retries'：1，1，'retry_delay'：datetime.timedelta（minutes = 5）， } schedule_interval = '45 09 * * *' dag = DAG（'xxx-merge'，default_args = default_args，schedule_interval = schedule_in时间段） hfan = DockerOperator（ task_id ='hfan'， image ='gcr.io/yyyyy/xxxx'）

...但是尝试运行时，它会在Web UI中告诉我：

破损的DAG：[/home/airflow/gcs/dags/xxxx.py]没有名为docker

也许Docker没有配置为在Cloud Composer运行的Kubernetes集群内工作吗？还是我只是缺少语法上的内容？

解决方案

我通过在以下位置安装docker-py == 1.10.6来解决它

但是，要使DockerOperator正常工作需要付出更多的努力，因为作曲者的工作人员无法访问Docker守护进程。转到GCP控制台并执行以下步骤；获得群集凭据）之后。

将当前的部署配置导出到文件中

kubectl获取部署流程- o yaml --export> airflow-worker-config.yaml

编辑airflow-worker-config.yaml（示例链接）将docker.socker和docker安装到运行中的docker-socker和docker权限

应用部署设置

kubectl apply -f airflow-worker- config.yaml

I have BigQuery connectors all running, but I have some existing scripts in Docker containers I wish to schedule on Cloud Composer instead of App Engine Flexible.

I have the below script that seems to follow the examples I can find:

import datetime from airflow import DAG from airflow import models from airflow.operators.docker_operator import DockerOperator yesterday = datetime.datetimebine( datetime.datetime.today() - datetime.timedelta(1), datetime.datetime.min.time()) default_args = { # Setting start date as yesterday starts the DAG immediately 'start_date': yesterday, # If a task fails, retry it once after waiting at least 5 minutes 'retries': 1, 'retry_delay': datetime.timedelta(minutes=5), } schedule_interval = '45 09 * * *' dag = DAG('xxx-merge', default_args=default_args, schedule_interval=schedule_interval) hfan = DockerOperator( task_id = 'hfan', image = 'gcr.io/yyyyy/xxxx' )

...but when trying to run it tells me in the web UI:

Broken DAG: [/home/airflow/gcs/dags/xxxx.py] No module named docker

Is it perhaps that the Docker is not configured to work inside the Kubernetes cluster that Cloud Composer runs? Or am I just missing something in the syntax?

解决方案

I got it resolved by installing docker-py==1.10.6 in the PyPI section of composer.

However, to get DockerOperator to work properly requires a bit more effort as the composer workers do not have access to the Docker daemon. Head to the GCP console and perform the following steps; after getting cluster credentials).

Export current deployment config to file

kubectl get deployment airflow-worker -o yaml --export > airflow-worker-config.yaml

Edit airflow-worker-config.yaml (example link) to mount docker.sock and docker, grant privileged access to airflow-worker to run docker commands

Apply deployment settings

kubectl apply -f airflow-worker-config.yaml