我有以下3个任务的DAG:
I have the following DAG with 3 tasks :
start --> special_task --> end中间任务可以成功或失败,但结束 必须总是被执行(想象这是清理资源的任务)。为此,我使用了触发规则 ALL_DONE :
The task in the middle can succeed or fail, but end must always be executed (imagine this is a task for cleanly closing resources). For that, I used the trigger rule ALL_DONE :
end.trigger_rule = trigger_rule.TriggerRule.ALL_DONE如果在以下情况下正确执行 end 特殊任务失败。但是,由于 end 是最后一个任务并且成功完成,因此DAG始终标记为 SUCCESS 。
Using that, end is properly executed if special_task fails. However, since end is the last task and succeeds, the DAG is always marked as SUCCESS.
如何配置DAG,以便如果其中一项任务失败,则整个DAG都标记为 FAILED ?
How can I configure my DAG so that if one of the tasks failed, the whole DAG is marked as FAILED?
import datetime from airflow import DAG from airflow.operators.bash_operator import BashOperator from airflow.utils import trigger_rule dag = DAG( dag_id='my_dag', start_date=datetime.datetime.today(), schedule_interval=None ) start = BashOperator( task_id='start', bash_command='echo start', dag=dag ) special_task = BashOperator( task_id='special_task', bash_command='exit 1', # force failure dag=dag ) end = BashOperator( task_id='end', bash_command='echo end', dag=dag ) end.trigger_rule = trigger_rule.TriggerRule.ALL_DONE start.set_downstream(special_task) special_task.set_downstream(end)这篇文章似乎相关,但是答案不符合我的需要,因为必须执行下游任务 end (因此,强制性 trigger_rule )。
This post seems to be related, but the answer does not suit my needs, since the downstream task end must be executed (hence the mandatory trigger_rule).
推荐答案为@JustinasMarozas 在评论,一种解决方案是创建一个虚拟任务,例如:
As @JustinasMarozas explained in a comment, a solution is to create a dummy task like :
dummy = DummyOperator( task_id='test', dag=dag )并将其下游绑定到 special_task :
failing_task.set_downstream(dummy)因此,DAG被标记为失败,并且 dummy 任务被标记为 upstream_failed 。
Thus, the DAG is marked as failed, and the dummy task is marked as upstream_failed.
希望在那里是一个开箱即用的解决方案,但请耐心等待,此解决方案可以完成工作。
Hope there is an out-of-the-box solution, but waiting for that, this solution does the job.
更多推荐
气流:DAG标记为“成功”;如果一项任务失败,由于触发规则ALL
发布评论