运行bashoperator时气流重置环境变量

本文介绍了运行bashoperator时气流重置环境变量的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

在执行一项气流任务时，我遇到了环境变量问题.

With one of my airflow task, I have an environment variable issue.

[2019-08-19 04:51:04,603] {{bash_operator.py:127}} INFO - File "/home/ubuntu/.pyenv/versions/3.6.7/lib/python3.6/os.py", line 669, in __getitem__ [2019-08-19 04:51:04,603] {{bash_operator.py:127}} INFO - raise KeyError(key) from None [2019-08-19 04:51:04,603] {{bash_operator.py:127}} INFO - KeyError: 'HOME' [2019-08-19 04:51:04,639] {{bash_operator.py:131}} INFO - Command exited with return code 1

我的任务如下:

task_name = BashOperator( task_id='task_name', bash_command="cd path/to/manage.py && export LC_ALL=C.UTF-8 && export LANG=C.UTF-8 " f'&& {Variable.get("python_virtualenv_path")}virtual-env-name/bin/python manage.py command_name', retries=1, pool='LightAndFast', dag=dag )

对这个问题有什么想法吗?

Any ideas of this issue?

推荐答案

这是事实，气流使用BashOperator重置环境变量，至少我遇到了这个问题.在操作员的文档中，可从以下位置获得: airflow.apache/docs/stable/_modules/airflow/operators/bash_operator.html ， 我找到了显式设置bash命令环境的方法，即

This is true that airflow resets environment variable, when using BashOperator, at least I faced this issue. In documentation of the operator, available at : airflow.apache/docs/stable/_modules/airflow/operators/bash_operator.html, I found the way to explicitly set the environment for the bash command i.e.

bash_task = BashOperator( task_id="bash_task", bash_command='echo "here is the message: \'$message\'"', env={'message': '{{ dag_run.conf["message"] if dag_run else "" }}'}, )

因此，我将Bash命令的环境明确设置为:

Hence I explicitly set the environment for the Bash command as:

env = os.environ.copy(),

请确保先在 dag 文件中选择import os.它为我解决了这个问题.

make sure to import os, earlier on in the dag file. And it resolved the issue for me.