尝试改为导入HtmlXPathSelector。

    from scrapy.selector import HtmlXPathSelector

然后使用.select（）方法解析出你的html。例如，

    sel = HtmlXPathSelector(response)
    site_names = sel.select('//ul/li')

示例将如下所示：

    from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector

    class DmozSpider(BaseSpider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
        ]

        def parse(self, response):
            sel = HtmlXPathSelector(response)
            sites = sel.select('//ul/li')

            for site in sites:
                title = site.select('a/text()').extract()
                link = site.select('a/@href').extract()
                desc = site.select('text()').extract()
                print title, link, desc

希望这可以帮助！