长时间超时的无服务器python请求?

本文介绍了长时间超时的无服务器python请求?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我有几个遵循类似格式的python脚本:您传入一个日期，并且它是:-检查我的S3存储桶中文件名中具有该日期的文件，并对其进行解析，或者-运行python脚本，对该日期的文件进行一些分析(运行需要1个小时以上)

I have a several python scripts that follow a similar format: you pass in a date, and it either: - checks my S3 bucket for the file with that date in the filename, and parses it or - Runs a python script doing some analysis on the file of that date (which take over 1 hour to run)

我正在寻找一种无服务器的解决方案，该解决方案可以让我在一定的日期范围内调用这些函数，然后并行运行它们.由于我的python脚本持续时间长，因此AWS和Google Cloud Functions之类的服务由于超时(分别为15分钟和9分钟)而无法正常工作.我已经看过Google Cloud Dataflow，但是不确定相对于我的相对简单的用例而言，这是否过分.

I am looking for a serverless solution that would let me call these functions on a range of dates, and run them all in parallel. Because of the long duration of my python script, services like AWS and Google Cloud Functions don't work because of their timeouts (15 minutes and 9 minutes respectively). I have looked at Google Cloud Dataflow, but am not sure whether this is overkill for my relatively simple use case.

尽可能减少停机的情况很重要，因此我倾向于使用AWS，Google Cloud等产品.

Something with the lowest possible outages is important, so I am leaning towards something from AWS, Google Cloud, etc.

我还希望能够通过日志查看每个作业进度的仪表板，因此我可以看到哪些日期已完成，哪些日期存在错误(以及错误是什么)

I also would like to be able to see a dashboard of the progress of each job with logs, so I can see which dates have completed and which dates had a bug (plus what the bug is)

推荐答案

Cloud Run路线图中已计划了长时间运行，但目前尚无日期.

Long running duration is planned in the Cloud Run roadmap but we don't have date for now.

今天，推荐的最佳方法是除任务外还使用AppEngine队列.使用推送队列，当您在手动缩放模式.但是请注意，手动缩放比例不会缩放为0！

Today, the best recommended way is to use AppEngine in addition of Task Queue. With push queue, you can run process up to 24H long when you deploy in manual scaling mode. But Be careful, manual scaling doesn't scale to 0!

如果您喜欢容器，我知道GCP上有2种奇怪"的解决方法:

If you prefer container, I know 2 "strange" workaround on GCP:

• 使用Cloud Build.通过Cloud Build，您可以在容器中构建自定义构建器.在容器中做任何您想做的事情，即使它不是用于构建某些东西.考虑为为处理步骤设置正确的超时时间.借助Cloud Build，您每天有120分钟的免费时间(在整个组织中共享，这不是每个项目的免费套餐！).您最多可以并行运行10个构建作业.
• 使用AI平台培训.与Cloud Build类似， AI平台培训允许您运行自定义容器进行处理，最初考虑进行培训.但是，它是一个容器，您可以在其中运行任何内容.这里没有免费套餐.您最多只能并行使用20个CPU，但您可以要求将限制增加到450个并发vCPU.

• Use Cloud Build. Cloud Build allows you to build custom builder in a container. Do whatever you want in this container, even if it's not for building something. Think to set up the correct timeout for your processing step. You have 120 minutes per day FREE with Cloud Build (shared across the entire organisation, it's not a free tier per project!). You can run up to 10 build jobs in parallel.
• Use AI Platform training. Similarly to Cloud Build, AI Platform training allows you to run a custom container for performing processing, initially think for training. But, it's a container, you can run whatever you want in it. No free tier here. You are limited to 20 CPU in parallel but you can ask for increasing the limit up to 450 concurrent vCPU.

可悲的是，它不像功能"或云运行"那样容易使用.您没有HTTP端点，只需用您想要和享受的日期来调用它即可.但是您可以将其包装到一个函数中，该函数执行对Cloud Build或AI Platform培训的API调用.

Sadly, it's not as easy as a Function or a Cloud Run to use. You don't have an HTTP endpoint and you simply call this with the date that you want and enjoy. But you can wrap this into a function which perform the API calls to the Cloud Build or the AI Platform training.