跨进程共享psycopg2/libpq连接

本文介绍了跨进程共享psycopg2/libpq连接的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

根据 psycopg2文档:

libpq连接不应使用，因此当使用诸如multiprocessing之类的模块或诸如FastCGI之类的分支Web部署方法时，请确保在 叉之后创建连接.

libpq connections shouldn’t be used by a forked processes, so when using a module such as multiprocessing or a forking web deploy method such as FastCGI make sure to create the connections after the fork.

跟随链接文档指向:

在Unix上，由于父进程和子进程共享相同的套接字和操作系统资源，因此分叉具有打开的libpq连接的进程可能导致无法预测的结果.因此，不建议使用这种用法，尽管从子进程执行exec来加载新的可执行文件是安全的.

On Unix, forking a process with open libpq connections can lead to unpredictable results because the parent and child processes share the same sockets and operating system resources. For this reason, such usage is not recommended, though doing an exec from the child process to load a new executable is safe.

但是看来打开插座的分叉过程固有的问题.那么psycopg2在打开连接时发出分叉警告的原因是什么?

But it seems there's no inherent problem with forking processes with open sockets. So what's the reason for psycopg2's warning against forking when connections are open?

我提出这个问题的原因是，我看到了一种(大概成功的)多处理方法，它在分叉之前就打开了连接.

The reason for my question is that I saw a (presumably successful) multiprocessing approach that opened a connection right before forking.

在某些限制下分叉打开的连接也许是安全的(例如，实际上只有一个进程使用该连接等)?

Perhaps it is safe to fork open connections under some restrictions (e.g., only one process actually ever uses the connection, etc.)?

推荐答案

您的猜测基本上是正确的:只要您不尝试在多个分支中使用它，在fork之前打开连接就没有问题.过程.

Your surmise is basically correct: there is no issue with a connection being opened before a fork as long as you don't attempt to use it in more than one process.

话虽如此，我认为您误解了您提供的多处理方法"链接.它实际上表明在每个孩子中打开了一个单独的连接. (在分叉之前，父级已打开一个连接，但未在任何子级中使用该连接.)

That being said, I think you misunderstood the "multiprocessing approach" link you provided. It actually demonstrates a separate connection being opened in each child. (There is a connection opened by the parent before forking, but it's not being used in any child.)

那里的答案(相对于问题中的代码)所带来的改进是重构，以便于-不是为队列中的每个任务打开新的连接-每个子进程都打开一个连接，然后在多个任务之间共享在同一子进程中执行(即，将连接作为参数传递给任务处理器).

The improvement given by the answer there (versus the code in the question) was to refactor so that -- rather than opening a new connection for each task in the queue -- each child process opened a single connection and then shared it across multiple tasks executed within the same child (i.e. the connection is passed as an argument to the Task processor).

修改: 通常，应该在使用它的过程中创建一个连接.在引用的答案中，在分叉之前先在父级中创建一个连接，然后在子级中使用该连接.这确实可以正常工作，但是每个子连接"也都在父级中保持打开状态，这充其量是在浪费资源和潜在错误的原因.

Edit: As a general practice, one should prefer creating a connection within the process that is using it. In the answer cited, a connection was being created in the parent before forking, then used in the child. This does work fine, but leaves each "child connection" open in the parent as well, which is at best a waste of resources and also a potential cause of bugs.