使用multiprocessing.Process并发进程数最多

本文介绍了使用multiprocessing.Process并发进程数最多的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我有Python代码:

from multiprocessing import Process def f(name): print 'hello', name if __name__ == '__main__': for i in range(0, MAX_PROCESSES): p = Process(target=f, args=(i,)) p.start()

运行良好.但是，MAX_PROCESSES是可变的，并且可以是1和512之间的任何值.由于我仅在具有8内核的计算机上运行此代码，因此我需要确定是否有可能限制允许同时运行的进程数.我已经研究过multiprocessing.Queue，但它看起来不符合我的需要-也许我对文档的解释不正确.

which runs well. However, MAX_PROCESSES is variable and can be any value between 1 and 512. Since I'm only running this code on a machine with 8 cores, I need to find out if it is possible to limit the number of processes allowed to run at the same time. I've looked into multiprocessing.Queue, but it doesn't look like what I need - or perhaps I'm interpreting the docs incorrectly.

是否可以限制同时运行multiprocessing.Process的次数?

Is there a way to limit the number of simultaneous multiprocessing.Processs running?

推荐答案

使用multiprocessing.Pool可能是最明智的选择，它会根据系统上可用的最大内核数生成一个工作进程池，然后进行基本操作核心成为可用的任务.

It might be most sensible to use multiprocessing.Pool which produces a pool of worker processes based on the max number of cores available on your system, and then basically feeds tasks in as the cores become available.

标准文档中的示例( docs. python/2/library/multiprocessing.html#using-a-pool-of-workers )显示，您还可以手动设置内核数:

The example from the standard docs (docs.python/2/library/multiprocessing.html#using-a-pool-of-workers) shows that you can also manually set the number of cores:

from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': pool = Pool(processes=4) # start 4 worker processes result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously print result.get(timeout=1) # prints "100" unless your computer is *very* slow print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"

如果代码中需要的话，知道multiprocessing.cpu_count()方法也可以方便地计算给定系统上的内核数.

And it's also handy to know that there is the multiprocessing.cpu_count() method to count the number of cores on a given system, if needed in your code.

以下一些代码草案似乎适用于您的特定情况:

Here's some draft code that seems to work for your specific case:

import multiprocessing def f(name): print 'hello', name if __name__ == '__main__': pool = multiprocessing.Pool() #use all available cores, otherwise specify the number you want as an argument for i in xrange(0, 512): pool.apply_async(f, args=(i,)) pool.close() pool.join()