Python中线程之间的信号传递(Signaling between threads in Python)

Python中线程之间的信号传递(Signaling between threads in Python) python

我正在研究实时数据采集器。 我有一段时间的True循环，在其中，我产生了执行相对较小任务的线程（我通过HTTP查询第三方API，并实现我并行查询的快速速度）。

每个线程都负责更新特定的数据系列。 这可能需要2秒，3秒甚至5秒。 但是，我的True循环可能比线程完成所花费的时间更快地生成线程。 因此，我需要生成的线程等待其先前的线程完成。

通常，由于线程查询HTTP服务器，线程完成所需的时间不可预测...

我想为每个线程创建一个命名信号量，然后如果为特定系列生成的线程发现前一个线程在同一个系列上工作，它将等待。

我能看到的唯一问题是可能积压的线程..

这里最好的解决方案是什么？ 我应该看看像芹菜这样的东西吗？ 我目前正在使用线程模块。

谢谢！

I am working on a realtime data grabber. I have a while True loop, and inside it, I spawn threads that do relatively small tasks (I am querying a 3rd party API over HTTP, and to achieve fast speeds I am querying in parallel).

Every thread takes care of updating a specific data series. This might take 2, 3 or even 5 seconds. However, my while True loop might spawn threads faster than how long it takes for the thread to finish. Hence, I need the spawned threads to wait for their previous threads to finish.

In general, its unpredictable how long it takes for the threads to finish because the threads query an HTTP server...

I was thinking of creating a named semaphore for every thread, and then if a thread spawned for a specific series finds a previous thread working on the same series, it will wait.

The only issue that I can see is a possible backlog of threads..

What is the best solution here? Should I look into things like Celery? I am currently using the threading module.

Thanks!

最满意答案

如果您在每次查询返回时只是重新查询API，那么您可以使用的另一个选项是像Twisted这样的异步框架（ 他们的线程教程 ）。 我是一个相对扭曲的初学者，所以可能有更好的方法扭曲扭曲到你的任务比这 -

from twisted.internet import reactor, defer def simple_task(): status = query_your_api() return status def repeating_call(status): print(status) d = threads.deferToThread(simple_task) d.addCallback(repeating_call) data_series = [data1, data2, data3] for data in data_series: repeating_call('starting everything up') reactor.run()

Another option you could use if you are just requerying the API every time one of your queries returns is an asyncronous framework like Twisted (Their tutorial on Threading). I'm a relative Twisted beginner so there may be better ways of twisting Twisted to your task than this -

from twisted.internet import reactor, defer def simple_task(): status = query_your_api() return status def repeating_call(status): print(status) d = threads.deferToThread(simple_task) d.addCallback(repeating_call) data_series = [data1, data2, data3] for data in data_series: repeating_call('starting everything up') reactor.run()