气流：任务排队但未运行

本文介绍了气流：任务排队但未运行的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我是气流的新手，正在尝试设置气流以运行ETL管道。我能够安装

I am new to airflow and trying to setup airflow to run ETL pipelines. I was able to install

气流

postgres

celery

rabbitmq

airflow

postgres

celery

rabbitmq

我能够测试运行turag dag。当我尝试调度作业时，调度程序可以将其拾取并排队，这些作业可以在UI上看到，但任务未运行。有人可以帮我解决这个问题吗？我相信我在这里缺少最基本的气流概念。以下是airflow.cfg

I am able to test run the turtorial dag. When i try to schedule the jobs, scheduler is able to pick it up and queue the jobs which i could see on the UI but tasks are not running. Could somebody help me fix ths issue? I believe i am missing most basic airflow concept here. below is the airflow.cfg

这是我的配置文件：

[core] airflow_home = /root/airflow dags_folder = /root/airflow/dags base_log_folder = /root/airflow/logs executor = CeleryExecutor sql_alchemy_conn = postgresql+psycopg2://xxxx.amazonaws:5432/airflow api_client = airflow.api.client.local_client [webserver] web_server_host = 0.0.0.0 web_server_port = 8080 web_server_worker_timeout = 120 worker_refresh_batch_size = 1 worker_refresh_interval = 30 [celery] celery_app_name = airflow.executors.celery_executor celeryd_concurrency = 16 worker_log_server_port = 8793 broker_url = amqp://rabbit:rabbit@x.x.x.x/rabbitmq_vhost celery_result_backend = db+postgresql+psycopg2://postgres:airflow@xxx.amazonaws:5432/airflow flower_host = 0.0.0.0 flower_port = 5555 default_queue = default

DAG：这是我使用的教程dag

，我的dag的开始日期是-'start_date'：datetime（2017，4，11），

and the start date for my dag is -- 'start_date': datetime(2017, 4, 11),

推荐答案

运行气流的所有三个组件，即：

have your run all the three components of airflow, namely:

airflow webserver airflow scheduler airflow worker

如果仅运行前两个，则任务将是已排队，但未执行。

If you only run the previous two, the tasks will be queued, but not executed. airflow worker will provide the workers that actually execute the dags.

此外，芹菜4.0.2目前与气流1.7或1.8不兼容。改用芹菜3。

Also btw, celery 4.0.2 is not compatible with airflow 1.7 or 1.8 currently. Use celery 3 instead.