使用Beanstalkd / Gearman在PHP中运行并发API请求(Running concurrent API requests in PHP using Beanstalkd/Gearman

使用Beanstalkd / Gearman在PHP中运行并发API请求(Running concurrent API requests in PHP using Beanstalkd/Gearman)

我有一个相当大的PHP Web应用程序，通过他们的API从许多其他供应商处获取其产品，通常用大量XML来解析。 目前有20家供应商，但这将进一步上升。

我们当前的设置使用多卷曲来发出请求，这需要大约30-40秒才能完成并且太长。 该脚本在后台运行，而前端轮询数据库以查找结果，然后在它们进入时显示它们。

为了改进这个过程，我们考虑使用作业服务器在后台运行，每个供应商请求都是一个单独的工作。 我们已经看到了beanstalkd和Gearman被提及。

那么，我们正在朝着正确的方向寻找正确的工作服务器吗？ 我们正在考虑尽快进行一些促销活动，以便我们可以让200多名用户同时搜索30家供应商，因此如果我们必须负载均衡，正确的选择需要扩展。

完全收到任何建议。

I've got a rather large PHP web app which gets its products from numerous others suppliers through their API's, usually responding with a large XML to parse. Currently there are 20 suppliers but this is due to rise even further.

Our current set up uses multi curl to make the requests and this takes about 30-40 seconds to complete and is too long. The script runs in the background whilst the front end polls the database looking for results and then displays them as they come in.

To improve this process we were thinking of using a job server to run in the background, each supplier request being a separate job. We've seen beanstalkd and Gearman being mentioned.

So are we looking in the right direction, as in, is a job server the right way to go? We're looking at doing some promotion soon so we may get 200+ users searching 30 suppliers at the same time so the right choice needs to scale well if we have to load balance.

Any advice is great fully received.

最满意答案

您可以使用Beanstalkd，因为您可以自定义作业的优先级和TTR的解析时间，默认为60秒，但对于您的方案，您必须增加它。 Beanstalkd有一个不错的管理控制台面板 。

您还应该利用多个Curl调用，因此您应该使用并行请求。 为了使用Keep-alive，您还需要维护一个CURL句柄池并保持温暖。 查看高性能卷曲提示 。 您还需要调整Linux网络堆栈 。

如果您在云中运行此功能，请确保使用多台微型计算机而不是一台重型计算机，因为当您有多个可用资源时，吞吐量会更好。

You can use Beanstalkd, as you can customize the priority of jobs and the TTR time-to-resolve, default is 60 seconds, but for your scenario you must increase it. There is a nice admin console panel for Beanstalkd.

You should also leverage the multi Curl calls, so you should use parallel requests. In order to make use of Keep-alive you also need to maintain a pool of CURL handles and keep them warm. See high performance curl tips. You also need to tune Linux network stack.

If you run this in cloud, make sure you use multiple micro machines rather than one heavy machine as the throughput is better when you have multiple resources available.