scrapy-redis

settings配置redis：

SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_HOST = '127.0.0.1'
REDIS_PORT = 6379

爬虫修改：

class NovelSpider(RedisSpider):
    name = 'novel2'
    redis_key = 'novel2:start_urls'
    start_urls = ['http://www.daomubiji.com/']