使用pyspark lib构建docker映像时出现问题

本文介绍了使用pyspark lib构建docker映像时出现问题的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我正在尝试使用s2i和Jenkins构建docker映像.我在requirement.txt文件

I'm trying to build a docker image using s2i and Jenkins. I have the following dependencies in the requirement.txt file

scikit-learn==0.21.2 scipy==0.18.1 pandas==0.24.2 seldon-core==0.3.0 pypandoc pyspark==2.4.1

但是当我尝试使用以下错误消息安装pyspark时，我的构建过程失败了

But my build process fails when it tries to install pyspark with the following error message

Downloading repopany/repository/pypi-all/packages/f2/64/a1df4440483df47381bbbf6a03119ef66515cf2e1a766d9369811575454b/pyspark-2.4.1.tar.gz (215.7MB) Complete output from command python setup.py egg_info: Could not import pypandoc - required to package PySpark Download error on pypi/simple/pypandoc/: [Errno 97] Address family not supported by protocol -- Some packages may not be found! Couldn't find index page for 'pypandoc' (maybe misspelled?) Download error on pypi/simple/: [Errno 97] Address family not supported by protocol -- Some packages may not be found! No local packages or working download links found for pypandoc Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-dra7nhke/pyspark/setup.py", line 224, in <module> 'Programming Language :: Python :: Implementation :: PyPy'] File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup _install_setup_requires(attrs) File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires ...

尽管我在requirments.txt文件中的pyspark之前列出了pypandoc，但是看起来像在安装pyspark时，尚未安装pypandoc.有什么问题吗?

although I have listed pypandoc before pyspark in the requirments.txt file, but it looks like when it's installing pyspark, pypandoc is not installed yet, which is a dependency. What's the problem?

推荐答案

我通过执行以下错误来传递此错误:

I passed this error by executing:

pip install pypandoc

在安装pyspark之前.我尝试了很多在Requires.txt中使用pypandoc == 1.4的方法，但是它不起作用.

before installing pyspark. I tried a lot use pypandoc==1.4 in requirements.txt but it doesn't work.

在这种情况下，人们使用相同的方式: hub.docker/r/takaomag/test-0/dockerfile www.ibm/support/Knowledgecenter/el/SSWTQQ_2.0.3/install/t_si_pythonpackagesoffline.html 无法安装pyspark

In that sources people using same way: hub.docker/r/takaomag/test-0/dockerfile www.ibm/support/knowledgecenter/el/SSWTQQ_2.0.3/install/t_si_pythonpackagesoffline.html unable to install pyspark