在Python中使用多重处理和请求进行并行发布请求

本文介绍了在Python中使用多重处理和请求进行并行发布请求的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我有如下的小代码段:

import requests import multiprocessing header = { 'X-Location': 'UNKNOWN', 'X-AppVersion': '2.20.0', 'X-UniqueId': '2397123', 'X-User-Locale': 'en', 'X-Platform': 'Android', 'X-AppId': 'com.my_app', 'Accept-Language': 'en-ID', 'X-PushTokenType': 'GCM', 'X-DeviceToken': 'some_device_token' } BASE_URI = 'my_server/v2/customers/login' def internet_resource_getter(post_data): stuff_got = [] response = requests.post(BASE_URI, headers=header, json=post_data) stuff_got.append(response.json()) return stuff_got tokens = [{"my_token":'EAAOZAe8Q2rKYBAu0XETMiCZC0EYAddz4Muk6Luh300PGwGAMh26Bpw3AA6srcxbPWSTATpTLmvhzkUHuercNlZC1vDfL9Kmw3pyoQfpyP2t7NzPAOMCbmCAH6ftXe4bDc4dXgjizqnudfM0D346rrEQot5H0esW3RHGf8ZBRVfTtX8yR0NppfU5LfzNPqlAem9M5ZC8lbFlzKpZAZBOxsaz'},{"my_token":'EAAOZAe8Q2rKYBAKQetLqFwoTM2maZBOMUZA2w5mLmYQi1GpKFGZAxZCaRjv09IfAxxK1amZBE3ab25KzL4Bo9xvubiTkRriGhuivinYBkZAwQpnMZC99CR2FOqbNMmZBvLjZBW7xv6BwSTu3sledpLSGQvPIZBKmTv3930dBH8lazZCs3q0Q5i9CZC8mf8kYeamV9DED1nsg5PQZDZD'}] pool = multiprocessing.Pool(processes=3) pool_outputs = pool.map(internet_resource_getter, tokens) pool.close() pool.join()

我要做的就是将并行的POST请求激发到终点，而每个POST都将具有不同的令牌作为其正文.

All I am trying to do is fire parallel POST requests to the end point, while each POST would have a different token as it's post body.

通过以上内容我将能够实现我想要的目标吗?我得到了输出，但是不确定我的请求是否并行发送.

我知道grequests.我想实现真正的并行请求(如利用系统上的多个处理器)，因此我选择了多处理而不是grequest(据我所知，它使用的是gevents，它又不是并行的，而是多线程的).我的理解在这里正确吗?

推荐答案

如果您对并行执行多个POST请求感兴趣，建议您使用 asyncio 或 aiohttp ，两者都实现了异步任务的概念，该任务可以并行运行.

If you interested in a parallel execution of multiple POST requests, I suggest you to use asyncio or aiohttp, which both implements the idea of asynchronous tasks, which run in parallel.

例如，您可以使用asyncio做这样的事情:

For example, you can do something like this with asyncio:

import requests import asyncio header = { 'X-Location': 'UNKNOWN', 'X-AppVersion': '2.20.0', 'X-UniqueId': '2397123', 'X-User-Locale': 'en', 'X-Platform': 'Android', 'X-AppId': 'com.my_app', 'Accept-Language': 'en-ID', 'X-PushTokenType': 'GCM', 'X-DeviceToken': 'some_device_token' } BASE_URI = 'my_server/v2/customers/login' def internet_resource_getter(post_data): stuff_got = [] response = requests.post(BASE_URI, headers=header, json=post_data) stuff_got.append(response.json()) print(stuff_got) return stuff_got tokens = [ { "my_token": 'EAAOZAe8Q2rKYBAu0XETMiCZC0EYAddz4Muk6Luh300PGwGAMh26B' 'pw3AA6srcxbPWSTATpTLmvhzkUHuercNlZC1vDfL9Kmw3pyoQfpyP' '2t7NzPAOMCbmCAH6ftXe4bDc4dXgjizqnudfM0D346rrEQot5H0es' 'W3RHGf8ZBRVfTtX8yR0NppfU5LfzNPqlAem9M5ZC8lbFlzKpZAZBO' 'xsaz' }, { "my_token": 'EAAOZAe8Q2rKYBAKQetLqFwoTM2maZBOMUZA2w5mLmYQi1GpKFGZAx' 'ZCaRjv09IfAxxK1amZBE3ab25KzL4Bo9xvubiTkRriGhuivinYBkZA' 'wQpnMZC99CR2FOqbNMmZBvLjZBW7xv6BwSTu3sledpLSGQvPIZBKmT' 'v3930dBH8lazZCs3q0Q5i9CZC8mf8kYeamV9DED1nsg5PQZDZD' } ] loop = asyncio.get_event_loop() for token in tokens: loop.run_in_executor(None, internet_resource_getter, token)

请注意:它们仅存在于python 3.x中.但是，我认为它看起来更好，更简洁，并且可以确保它们并行运行.

Note this: They only exist in python 3.x. But, it's look much better and concise in my opinion, and it's insure that they run in parallel.