multiprocessing.Manager 嵌套共享对象不适用于队列

本文介绍了multiprocessing.Manager 嵌套共享对象不适用于队列的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

Python docs of the multiprocessing module state:

Changed in version 3.6: Shared objects are capable of being nested. For example, a shared container object such as a shared list can contain other shared objects which will all be managed and synchronized by the SyncManager.

This does work with list and dict. However, if I try to create a shared Queue inside a shared dict, I get an error:

>>> from multiprocessing import Manager >>> m = Manager() >>> d = m.dict() >>> d['a'] = m.list() >>> d['b'] = m.dict() >>> d['c'] = m.Queue() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 2, in __setitem__ File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) multiprocessing.managers.RemoteError: --------------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client request = recv() File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv return _ForkingPickler.loads(buf.getbuffer()) File "/usr/lib/python3.6/multiprocessing/managers.py", line 881, in RebuildProxy return func(token, serializer, incref=incref, **kwds) TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned' ---------------------------------------------------------------------------

Seems like hg.python/cpython/rev/39e7307f9aee is the changeset which introduced nested shared objects.

解决方案

The error is caused by AutoProxy currently not handling all BaseProxy arguments. There's a pull request which has not been merged yet. You either need to monkey-patch AutoProxy, or you look into multiprocessing.managers.py and apply the changes in the patch here directly to your source code.

It's really important to fix both lines in the patch to prevent a memory leak in the server process. The manager_owned-flag is used to let the BaseProxy code know, when to skip a reference increment for a proxy the manager owns himself (through nesting).