我正在使用scrapy 1.1。我想添加scrapy-fake-user-agentScrapy中间件,该中间件将无缝且随机地轮换用户代理。用户代理字符串由fake-useragent模块提供。
scrapy-fake-user-agentScrapy
fake-useragent
按照网站的指示,我有:
DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, }
但是,当我运行它时,我得到:
ImportError: No module named scrapy_fake_useragent.middleware
我究竟做错了什么?
编辑:我在virtualenv中打开了一个python shell:
In[4]: import scrapy.downloadermiddlewares as x In[5]: x.useragent.UserAgentMiddleware Out[5]: scrapy.downloadermiddlewares.useragent.UserAgentMiddleware In[7]: import scrapy_fake_useragent as x In[8]: dir(x) Out[8]: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
因此,“ scrapy.downloadermiddlewares.useragent.UserAgentMiddleware”似乎可用,但没有“ scrapy_fake_useragent.middleware.RandomUserAgentMiddleware”吗?
我也有同样的问题。后来我发现我不在virutalenv内部。这些软件包已安装在virutalenv中。当移入virtualenv时,就可以了。