Scrapy 是单线程还是多线程?

本文介绍了Scrapy 是单线程还是多线程?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

Scrapy 中的并发设置很少，例如 CONCURRENT_REQUESTS.这是否意味着 Scrapy 爬虫是多线程的?因此，如果我运行 scrapy crawl my_crawler 它实际上会并行触发多个并发请求?我问是因为，我读过 Scrapy 是单线程的.

There are few concurrency settings in Scrapy, like CONCURRENT_REQUESTS. Does it mean, that Scrapy crawler is multi-threaded? So if I run scrapy crawl my_crawler it will literally fire multiple simultaneous requests in parallel? Im asking because, I've read that Scrapy is single-threaded.

推荐答案

Scrapy 是单线程的，除了交互式 shell 和一些测试，参见 来源.

Scrapy is single-threaded, except the interactive shell and some tests, see source.

它建立在 Twisted 之上，它也是单线程的，并利用了它有自己的异步并发功能，例如 twisted.internet.interfaces.IReactorThreads.callFromThread，见来源.

It's built on top of Twisted, which is single-threaded too, and makes use of it's own asynchronous concurrency capabilities, such as twisted.internet.interfaces.IReactorThreads.callFromThread, see source.