为什么不能在multiprocessing.Pool中使用operator.itemgetter?

本文介绍了为什么不能在multiprocessing.Pool中使用operator.itemgetter?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

以下程序:

import multiprocessing,operator f = operator.itemgetter(0) # def f(*a): return operator.itemgetter(0)(*a) if __name__ == '__main__': multiprocessing.Pool(1).map(f, ["ab"])

失败，并出现以下错误:

fails with the following error:

Process PoolWorker-1: Traceback (most recent call last): File "/usr/lib/python3.2/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/usr/lib/python3.2/multiprocessing/process.py", line 116, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.2/multiprocessing/pool.py", line 102, in worker task = get() File "/usr/lib/python3.2/multiprocessing/queues.py", line 382, in get return recv() TypeError: itemgetter expected 1 arguments, got 0

为什么会出现错误(在Linux x64上的cPython 2.7和3.2上)，如果我取消注释第三行，为什么它消失了?

Why do I get the error (on cPython 2.7 and 3.2 on Linux x64), and why does it vanish if I uncomment the third line?

推荐答案

这里的问题是，多处理模块通过复制将对象传递给其他进程(很明显)，并且itemgetter对象无法使用任何明显的方式进行复制:

The problem here is that the multiprocessing module passes objects by copy into the other processes (obviously), and itemgetter objects are not copyable using any of the obvious means:

In [10]: a = operator.itemgetter(0) Out[10]: copy.copy(a) TypeError: itemgetter expected 1 arguments, got 0 In [10]: a = operator.itemgetter(0) Out[10]: copy.deepcopy(a) TypeError: itemgetter expected 1 arguments, got 0 In [10]: a = operator.itemgetter(0) Out[10]: pickle.dumps(a) TypeError: can't pickle itemgetter objects # etc.

问题甚至没有尝试在其他进程中调用f；首先尝试复制它. (如果您查看我在上面省略的堆栈跟踪，则会看到很多有关失败原因的信息.)

The problem isn't even attempting to call f inside the other processes; it's trying to copy it in the first place. (If you look at the stack traces, which I omitted above, you'll see a lot more information on why this fails.)

当然，通常这无关紧要，因为快速构造新的itemgetter和复制一个itemgetter几乎既简单又有效.这就是您替代的"f"函数正在执行的操作. (当然，复制动态创建itemgetter的函数不需要复制itemgetter.)

Of course usually this doesn't matter, because it's nearly as easy and efficient to construct a new itemgetter on the fly as to copy one. And this is what your alternative "f" function is doing. (Copying a function that creates an itemgetter on the fly doesn't require copying an itemgetter, of course.)

您可以将"f"转换为lambda.或编写一个琐碎的函数(名为lambda)，而无需使用itemgetter即可执行相同的操作.或编写一个可复制的itemgetter替代品(显然不会那么难).但是您不能像您想要的那样直接使用itemgetter对象.

You could turn "f" into a lambda. Or write a trivial function (named or lambda) that does the same thing without using itemgetter. Or write an itemgetter replacement that is copyable (which obviously wouldn't be all that hard). But you can't directly use itemgetter objects as-is the way you want to.