Python:使用request重爬Python教程

最近在使用requests库之后，简直拜服这个库啊，简直是神器。跟select方法配合使用，无往而不胜。

几个CSS tips：

1. 如果button上有class属性的，如:

可以这样写：

css=button.x-right-button

.代表class

2. 如果class里带的空格，用.来代替空格如：

...

可以这样写：

css=button.x-btn-text.module_picker_icon

3. 如果想用别的属性值定位，用方括号【属性名=属性值】的方式，如：

Abc
css=abbr[title="Abc"]
```
4. 如果button上有id属性的，如：
```

```
可以这样写：
```
css=input#ag_name 
```
或者这样
```
css=#ag_name
#代表id
```

详细分析可见[这里](http://www.jianshu.com/p/8441de1c2f03)
具体代码如下：
```
#!usr/bin/env  
# -*-coding:utf-8 -*-
import requests
from bs4 import BeautifulSoup as BS
import os
import sys

reload(sys)
sys.setdefaultencoding('utf-8')

sub_folder = os.path.join(os.getcwd(), "liaoxuefeng")
if not os.path.exists(sub_folder):
    os.mkdir(sub_folder)

#you may not need this
proxies = {
  "http": "http://proxy.com:8080/",
  "https": "https://proxy.com:8080/",
}

domain = 'http://www.liaoxuefeng.com'
python27_url = domain + '/wiki/001374738125095c955c1e6d8bb493182103fac9270762a000'

web_data = requests.get(python27_url, proxies=proxies)
soup = BS(web_data.text, "lxml")

link_lists = soup.select("div.x-sidebar-left-content > ul.uk-nav.uk-nav-side > li > a")

article_maps = []
for article in link_lists:
    article_maps.append(article.get("href"))

for i in range(len(article_maps)):
    url_of_each_chapter = domain + article_maps[i]
    # print url_of_each_chapter
    resp = requests.get(url=url_of_each_chapter, proxies=proxies)
    soup = BS(resp.text, "lxml")
    titles = soup.select("div.x-content > h4")

    title_of_each_chapter = ""
    for title in titles:
        title_of_each_chapter += title.get_text().decode('utf-8').replace("/", " ")
    # print title_of_each_chapter
    contents = soup.select("div.x-wiki-content > p")
    content_of_each_chapter = ""
    for content in contents:
        content_of_each_chapter += content.get_text() + "\n"
    filename = sub_folder + "\\" + str(i) + title_of_each_chapter + ".html"
    # print filename
    with open(filename, "wb") as f:
        f.write(content_of_each_chapter)
    f.close()
```