在 Python 3.5 中使用 aiohttp 获取多个 url

本文介绍了在 Python 3.5 中使用 aiohttp 获取多个 url的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

自从 Python 3.5 引入 async with 以来，文档 中推荐的语法a> aiohttp 已更改.现在获取他们建议的单个网址:

Since Python 3.5 introduced async with the syntax recommended in the docs for aiohttp has changed. Now to get a single url they suggest:

import aiohttp import asyncio async def fetch(session, url): with aiohttp.Timeout(10): async with session.get(url) as response: return await response.text() if __name__ == '__main__': loop = asyncio.get_event_loop() with aiohttp.ClientSession(loop=loop) as session: html = loop.run_until_complete( fetch(session, 'python')) print(html)

如何修改它以获取一组网址而不是一个网址?

How can I modify this to fetch a collection of urls instead of just one url?

在旧的 asyncio 示例中，您将设置任务列表，例如

In the old asyncio examples you would set up a list of tasks such as

tasks = [ fetch(session, 'cnn'), fetch(session, 'google'), fetch(session, 'twitter') ]

我尝试将这样的列表与上述方法结合起来，但失败了.

I tried to combine a list like this with the approach above but failed.

推荐答案

对于并行执行，您需要一个 异步任务

For parallel execution you need an asyncio.Task

我已将您的示例转换为从多个来源获取并发数据:

I've converted your example to concurrent data fetching from several sources:

import aiohttp import asyncio async def fetch(session, url): async with session.get(url) as response: if response.status != 200: response.raise_for_status() return await response.text() async def fetch_all(session, urls): tasks = [] for url in urls: task = asyncio.create_task(fetch(session, url)) tasks.append(task) results = await asyncio.gather(*tasks) return results async def main(): urls = ['cnn', 'google', 'twitter'] async with aiohttp.ClientSession() as session: htmls = await fetch_all(session, urls) print(htmls) if __name__ == '__main__': asyncio.run(main())