我在AWS中有一个相对简单的线性回归lambda.每个实例中称为函数的日志显示以下内容:
I have a relatively simple linear regression lambda in AWS. Each instance the function is called the logs display the following:
/opt/python/sklearn/externals/joblib/_multiprocessing_helpers.py:38: UserWarning: [Errno 38] Function not implemented. joblib will operate in serial modewarnings.warn('%s. joblib will operate in serial mode' % (e,))
我怀疑这是由于sklearn在lambda(即无服务器")上运行,并试图根据此问题和此GH问题.
I suspect this is due to sklearn running on a lambda (i.e. 'serverless') and trying to determine it's multi-processing capabilities as per this question and this GH issue.
我还从GH那里了解到这不是一个可修复"的问题,当在这些硬件上具有这些依赖项进行部署时,总是会发生这种情况.我恢复了预期的结果(即使我当前已将默认的最小lambda内存最大化为128mb).
I am also understanding from the GH that this is not a 'fixable' issue, it will always happen when deploying with these dependencies on this hardware. I am getting back my expected results (even though I am currently maxing out the default, minimum lambda memory of 128mb).
我的目标是控制警告,并且会知道是否有办法进行以下操作:
I aim to control the warnings and would know if there is a way to either:
- 停止sklearn寻找多处理程序,以防止发出警告
- 捕获此特定警告,并防止其从我的函数传递到cloudwatch日志中
- 如果两者都可行,那么从aws体系结构/pythonic观点来看这是更可取的吗?
- stop sklearn looking for multiprocessing, so preventing the warning from issuing
- capture this specific warning and prevent it from being passed from my function into the cloudwatch logs
- if both are possible, which would be preferable from a aws architecture/pythonic opinion?
要捕获警告并防止其传递到cloudwatch日志中,可以按以下方式过滤警告.
To capture the warning and prevent it from being passed into the cloudwatch logs, you can filter the warning as follows.
import json import warnings warnings.filterwarnings('error') try: import sklearn except Warning: pass def lambda_handler(event, context): # TODO implement return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }此处的文章,尤其是最后,重新创建并过滤警告.
The article here, particularly towards the end, recreates and filters the warning.
更多推荐
我应该如何在AWS Lambda实现中对待joblib多处理?
发布评论