如何使用asyncio和aiohttp异步通过API响应进行分页

本文介绍了如何使用asyncio和aiohttp异步通过API响应进行分页的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我正在尝试使用python异步进行api调用。我在列表中有多个端点，每个端点将返回分页结果。我可以通过多个端点进行异步设置，但是无法返回每个端点的分页结果。

I'm trying to make api calls with python asynchronously. I have multiple endpoints in a list and each endpoint will return paginated results. I'm able to set up going though the multiple endpoints asynchronously, however am not able to return the paginated results of each endpoint.

从调试中，我发现 fetch_more（）函数运行while循环，但实际上并没有得到通过session.get（）异步。所以基本上。函数 fetch_more（）旨在从api调用中获取每个端点的其余结果，但是我发现无论是否包含 fetch_more（）函数。我曾尝试寻找与asyncio分页的示例，但运气不太好。

From debugging, I found that the fetch_more() function runs the while loop, but doesn't actually get past the async with session.get(). So basically. The function fetch_more() is intended to get remaining results from the api call for each endpoint, however I find that the same number of results are returned with or without the fetch_more() function. I've tried looking for examples of pagination with asyncio but have not have much luck.

据我了解，我不应该在while循环内进行请求，但是，我不确定为获得分页结果而应该采取的方法。

From my understanding, I should not be doing a request inside a while loop, however, I'm not sure a way around that in order to get paginated results.

if __name__ == 'main': starter_fun(url, header, endpoints): starter_func(url, header, endpoints): loop = asyncio.get_event_loop() #event loop future = asyncio.ensure_future(fetch_all(url, header, endpoints)) loop.run_until_complete(future) #loop until done async def fetch_all(url, header, endpoints): async with ClientSession() as session: for endpoint in endpoints: task = asyncio.ensure_future(fetch(url, header, endpoint)) tasks.append(task) res = await asyncio.gather(*tasks) # gather task responses return res async def fetch(url, header, endpoint): total_tasks = [] async with session.get(url, headers=header, params=params, ssl=False) as response: response_json = await response.json() data = response_json['key'] tasks = asyncio.ensure_future(fetch_more(response_json, data, params, header, url, endpoint, session)) //this is where I am getting stuck total_tasks.append(tasks) return data //function to get paginated results of api endpoint async def fetch_more(response_json, data, params, header, url, endpoint, session): //this is where I am getting stuck while len(response_json['key']) >= params['limit']: params['offset'] = response_json['offset'] + len(response_json['key']) async with session.get(url, headers=header, params=params, ssl=False) as response_continued: resp_continued_json = await response_continued.json() data.extend(resp_continued_json[kebab_to_camel(endpoint)]) return data

当前我是使用或不使用 fetch_more 函数可以得到1000个结果，但是使用 fetch_more 可以得到更多的结果。有关如何异步分页的任何想法？

Currently I am getting 1000 results with or without the fetch_more function, however it should be a lot more with the fetch_more. Any idea as to how to approach asynchronously paginating?

推荐答案

from aiohttp import web async def fetch(self size: int = 10): data = "some code to fetch data here" def paginate(_data, _size): import itertools while True: i1, i2 = itertools.tee(_data) _data, page = (itertools.islice(i1, _size, None), list(itertools.islice(i2, _size))) if len(page) == 0: break yield page return web.json_response(list(paginate(_data=data, _size=size)))