aiohttp设置每秒请求数

本文介绍了aiohttp设置每秒请求数的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我正在Flask中编写一个API，其中包含1000多个获取数据的请求，我想限制每秒的请求数.我尝试过:

I'm writing an API in Flask with 1000+ requests to get data and I'd like to limit the number of requests per second. I tried with:

conn = aiohttp.TCPConnector(limit_per_host=20) and conn = aiohttp.TCPConnector(limit=20)

但是似乎不起作用

我的代码如下:

import logging import asyncio import aiohttp logging.basicConfig(filename="logfilename.log", level=logging.INFO, format='%(asctime)s %(levelname)s:%(message)s') async def fetch(session, url): async with session.get(url, headers=headers) as response: if response.status == 200: data = await response.json() json = data['args'] return json async def fetch_all(urls, loop): conn = aiohttp.TCPConnector(limit=20) async with aiohttp.ClientSession(connector=conn, loop=loop) as session: results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True) return results async def main(): loop = asyncio.new_event_loop() url_list = [] args = ['a', 'b', 'c', +1000 others] urls = url_list for i in args: base_url = 'httpbin/anything?key=%s' % i url_list.append(base_url) htmls = loop.run_until_complete(fetch_all(urls, loop)) for j in htmls: key = j['key'] # save to database logging.info(' %s was added', key)

如果我运行代码，则在1秒内我发送了200多个请求.有什么办法可以限制请求?

If I run code, within 1s I send over than 200 requests. Is there any way to limit requests?

推荐答案

上面的代码按预期工作(除了有关 headers 未定义的小错误).

The code above works as expected (apart from a small error regarding headers being undefined).

在我的计算机上进行测试， httpbin URL大约在100毫秒内响应，这意味着并发性为20的URL将在1秒内处理大约200个请求(这也是您所看到的):

Tested on my machine the httpbin URL responds in around 100ms which means that with a concurrency of 20 it will serve around 200 requests in 1 second (which is what you're seeing as well):

100毫秒表示 10个请求在一秒钟内完成每秒 10个请求和并发性 20 表示在一秒钟内 200个请求

100 ms per request means 10 requests are completed in a second 10 requests per second with a concurrency of 20 means 200 requests in one second

限制选项( aiohttp.TCPConnector )限制并发请求的数量，并且没有任何时间维度.

The limit option (aiohttp.TCPConnector) limits the number of concurrent requests and does not have any time dimension.

要查看实际限制，请尝试使用更多值，例如 10 ， 20 ， 50 :

To see the limit in action try with more values like 10, 20, 50:

# time to complete 1000 requests with different keys aiohttp.TCPConnector(limit=10): 12.58 seconds aiohttp.TCPConnector(limit=20): 6.57 seconds aiohttp.TCPConnector(limit=50): 3.1 seconds

如果您想使用每秒 requests 的限制来发送一批请求(例如20个)，然后使用 asyncio.sleep(1.0)暂停一秒钟，然后发送下一批，依此类推.

If you want to use a requests per second limit send batch of requests (20 for example) and use asyncio.sleep(1.0) to pause for a second, then send the next batch and so on.