Scrapy中报错URLWarning: allowed

报错URLWarning: allowed"/>

Scrapy中报错URLWarning: allowed

现象

源代码如下

class HrSpider4Spider(CrawlSpider):"""CrawlSpider类"""name = 'hr_spider4'allowed_domains = ['']  # 留意此处是一个完整的URL地址start_urls = [".php?&start=0"]rules = (Rule(LinkExtractor(allow=r'position.php\?&start=\d+')), # 默认follow参数为True,表示继续提取Rule(LinkExtractor(allow=r'position_detail\.php\?id=\d+'), callback="parse_item", follow=False))def parse_item(self, response):item = PositionItem()item['position_duty'] = response.xpath("//ul[@class='squareli']")[0].xpath(".//li/text()").extract()item['position_duty'] = response.xpath("//ul[@class='squareli']")[1].xpath(".//li/text()").extract()yield item

在运行该爬虫的时候会报错:
URLWarning: allowed_domains accepts only domains, not URLs.
原因显而易见: 允许范围接收的是范围, 而非URL地址.

解决方法

将第4行代码修改为

allowed_domains = ['hr.tencent'] 

也就是仅保留后缀.