安装 Scrapy for Python2.7

1. 简介

Scrapy是基于Python的网络爬虫框架，支持Python2.7 和 Python3.3以上。

2. 安装

安装平台： Mac 10.11

2.1 先决条件

• python 2.7
• pip
• setuptools
• lxml
• openssl

2.2 安装scrapy

pip install scrapy

2.3 重新安装six

Mac自带的six库需要重新安装

sudo rm -rf /Library/Python/2.7/site-packages/six*
sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six*
sudo pip install six

2.4 重新安装twisted

安装scrapy时自动安装的twisted框架版本有问题，需要重新安装

pip install twisted==13.1.0

3. 使用

现在就能使用scrapy了

$ scrapy
Scrapy 1.3.3 - project: tutorial

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  check         Check spider contracts
  commands      
  crawl         Run a spider
  edit          Edit spider
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  list          List available spiders
  parse         Parse URL (using its spider) and print the results
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

Use "scrapy  -h" to see more info about a command

4. 相关链接

中文教程： http://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/install.html#intro-install