如何让Mixnode Crawler爬得更慢？(How do I make the Mixnode Crawler to crawl slower?)

如何让Mixnode Crawler爬得更慢？(How do I make the Mixnode Crawler to crawl slower?)

我们突破2000万页/小时，我真的很欣赏速度; 但是我担心我可能会对目标网站施加太大的压力，我们有什么方法可以降低网站被抓取的速度？

We topped 20 million pages/hour and I truly appreciate the speed; however I'm afraid I may be putting too much pressure on target sites, is there any way we can decrease the speed at which websites are crawled?

最满意答案

不确定为什么你想要降低速度，因为文档明确指出：

发送到同一网站的请求之间至少有10秒的延迟 。 如果网站的robots.txt指令需要更长的延迟，Mixnode将遵循robots.txt指令指定的延迟时间。

Not sure why you'd want to decrease the speed as the documentation clearly states that :

There is a minimum delay of 10 seconds between requests sent to the same website. If robots.txt directives of a website require a longer delay, Mixnode will follow the delay duration specified by the robots.txt directives.