挑战:如何发送 >使用 Python 在一秒内处理 1000 个 HTTP 请求

本文介绍了挑战:如何发送 >使用 Python 在一秒内处理 1000 个 HTTP 请求的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

考虑以下情况:有一个缓慢的服务器，它使用大约 200 毫秒来处理请求(不包括网络传输时间).现在，我们需要每秒发送一堆请求.

Consider the following case: There is a slow server which use about 200ms to handle a request (no including the network transfer time). And now, we need to send a bunch of requests every second.

阅读本文后post，我试过多线程、多进程、twisted(agent.request)和eventlet.但是最大的加速只有6x，这是通过twisted和eventlet实现的，两者都使用epoll.

After read this post, I have tried multi-thread, multi-process, twisted (agent.request) and eventlet. But the biggest speedup is only 6x, which is achieved via twisted and eventlet, both are using epoll.

以下代码展示了带有 eventlet 的测试版本，

The following code shows the test version with eventlet,

import eventlet
eventlet.monkey_patch(all=False, socket=True)

import requests

def send():
    pile = eventlet.GreenPile(30)
    for i in range(1000):
        pile.spawn(requests.get, 'https://api.???/', timeout=1)
    for response in pile:
        if response:
            print response.elapsed, response.text

谁能帮我弄清楚为什么加速如此之低?还有没有其他机制可以让它更快?

Anyone could help me to make it clear why the speedup is so low? And is there any other mechanism could make it much faster?

推荐答案

我知道这是一个旧帖子，但可能仍然有人需要这个.

I know this is an old post but someone might still need this.

如果你想做负载测试但又想使用python，那么你应该使用像locust这样的工具:http://locust.io/

If you want to do load testing but want to use python then you should use a tool like locust: http://locust.io/

这是我在 10 秒内产生 10,000 个请求的解决方案:

Here is my solution which resulted in 10,000 requests in 10 seconds:

所需的包裹:须藤 pip 安装 grequests

Needed Package: sudo pip install grequests

代码:

import grequests
import time

start_time = time.time()
# Create a 10000 requests
urls = ['http://www.google.co.il']*10000
rs = (grequests.head(u) for u in urls)

# Send them.
grequests.map(rs)

print time.time() - start_time # Result was: 9.66666889191

这篇关于挑战:如何发送 >使用 Python 在一秒内处理 1000 个 HTTP 请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！