Docker上的气流

本文介绍了Docker上的气流-路径问题的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

使用气流，我尝试简单的DAG工作.

Working with airflow I try simple DAG work.

我编写了自定义运算符和其他要导入到DAG逻辑所在的主文件中的文件.

I wrote custom operators and other files that I want to import into the main file where the DAG logic is.

这里是文件夹的结构:

├── airflow.cfg ├── dags │   ├── __init__.py │   ├── dag.py │   └── sql_statements.sql ├── docker-compose.yaml ├── environment.yml └── plugins ├── __init__.py └── operators ├── __init__.py ├── facts_calculator.py ├── has_rows.py └── s3_to_redshift.py

我直接在 compose文件中设置了卷，因为当我登录到容器的终端时可以看到它们.

I setup the volume right in the compose file since I can see them when I log into the container's terminal.

我在线上关注了一些教程，在其中添加了一些 __ init __.py .

I followed some tutorials online from where I have added some __init__.py.

两个空的 __ init __ 都放入

• /plugins/operators :

from operators.facts_calculator import FactsCalculatorOperator from operators.has_rows import HasRowsOperator from operators.s3_to_redshift import S3ToRedshiftOperator __all__ = [ 'FactsCalculatorOperator', 'HasRowsOperator', 'S3ToRedshiftOperator' ]

• /插件:
from airflow.plugins_manager import AirflowPlugin import operators # Defining the plugin class class CustomPlugin(AirflowPlugin): name = "custom_plugin" # A list of class(es) derived from BaseOperator operators = [ operators.FactsCalculatorOperator, operators.HasRowsOperator, operators.S3ToRedshiftOperator ] # A list of class(es) derived from BaseHook hooks = [] # A list of class(es) derived from BaseExecutor executors = [] # A list of references to inject into the macros namespace macros = [] # A list of objects created from a class derived # from flask_admin.BaseView admin_views = [] # A list of Blueprint object created from flask.Blueprint flask_blueprints = [] # A list of menu links (flask_admin.base.MenuLink) menu_links = []

但是我不断从我的IDE中收到错误消息(说在操作符的 __ init __ 中没有名为操作符的模块或未解析的引用操作符).

But I keep getting errors from my IDE (saying No module named operators or Unresolved reference operators inside the operator's __init__).

由于一切都无法在网络服务器上启动.

Since everything fails to launch on the webserver.

任何想法如何进行设置?我哪里错了?

Any idea how to set this up ? Where I'm wrong ?

推荐答案

您是否正在使用 puckel的图片?

如果是，则需要取消注释docker-compose文件中的#-./plugins:/usr/local/airflow/plugins 行(可能有多个)本地或芹菜).其余的设置对我来说都很好.

If you are, you need to uncomment the # - ./plugins:/usr/local/airflow/plugins lines (may there are more than one) in the docker-compose files (either Local or Celery). The rest of your setup looks fine to me.