Python,PHP模拟浏览器发送HTTP请求

最近研究微博，想取自己的微薄RSS订阅Feed，但在国内（包括sina微博），都不提供个人微博的RSS。要取得自己的微博内容便于站外展示，只有通过官方发布的挂件 (sina , QQ)

• sina取个人RSS的方案可以参考：月光博客。原理是抓取sina的挂件，然后解析HTML 获得。

• 腾讯微薄就比较麻烦了，虽然他也有挂件，蛋，如果直接抓iframe的地址发现不行。查看挂件HTTP请求，发现最后会请求一个地址，如下：

  http://v.t.qq.com/cgi-bin/weiboshow?f=p&tweetflag=1&fansflag=0&fansnum=0&
  name=bbayou&sign=7bf0ab71321a3f376b710d2c739a2a8788ee1b0f

  然后返回一个json数据，腾讯服务器会有refer验证，也就是说只支持rerfer地址为 v.t.qq.com 的HTTP请求，直接访问会报错。所以，我们可以通过自定义 header refer，来访问接口。

  这里需要注意的是,参数：name (腾讯围脖用户名) , sign (腾讯围脖sign码，是唯一的)

取腾讯微博RSS，python代码如下，GET

qqurl = "/cgi-bin/weiboshow?f=p&tweetflag=1&fansflag=0&fansnum=0&name=bbayou&sign=7bf0ab71321a3f376b710d2c739a2a8788ee1b0f";

import httplib

headers = {"Cache-Control": "no-cache","Referer":"http://v.t.qq.com/"}

conn = httplib.HTTPConnection("v.t.qq.com")

conn.request("GET", qqurl, '', headers)

response = conn.getresponse()

print response.status, response.read()

conn.close();

如果腾讯需要POST，如下，注意，POST的时候，尽量把header信息填全，不然有时会通不过服务器验证

import httplib,urllib; 

params = urllib.urlencode({'cnum':'20','ctype':'CN02','isform':'true'})

headers = {"Content-Type":"application/x-www-form-urlencoded","Referer":"http://v.t.qq.com/"};

conn = httplib.HTTPConnection("v.t.qq.com")

conn.request(method="POST",url="/cgi-bin/weiboshow",body=params,headers=headers)

response = conn.getresponse()

print response.status, response.read()

conn.close()

因为在wordpress上使用，不支持python，写了个PHP，这里用Curl比较靠谱一点，fsockopen不太稳定

<?php

class QQClass

{

	var $Server = 'v.t.qq.com';

	var $Host = 'localhost';

	var $header = '';  

	var $Get_QQpath;

	var $Referer = 'http://v.t.qq.com/';

	var $Content = '';	

	var $Fp = '';

	function __construct($url){

		$this->Get_QQpath = $url;

		$this->Fp = fsockopen($this->Server,80, $errno, $errstr, 30);

		if (!$this->Fp){

			echo "ERROR: ".$errno." - ".$errstr."<br />\n";

			exit;

		}

		$out = "GET ".$this->Get_QQpath." HTTP/1.0\r\n";

		$out .= "Host: ".$this->Host." \r\n";

		$out .= "Referer: ".$this->Referer."\r\n";

		$out .= "Connection: Close\r\n\r\n";

		$out .=	"Accept-Language	zh-cn,zh;q=0.5";

		$out .=	"Cache-Control	no-cache";

		$out .=	"User-Agent	Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.0.19) Gecko/2010031422 Firefox/3.0.19 GTB7.1";

		$this->header = $out;

	}

	function Get_json(){

		if(function_exists('curl_init')){	

			$curl = curl_init();    

			curl_setopt($curl, CURLOPT_URL, $this->Get_QQpath);

			curl_setopt($curl, CURLOPT_CUSTOMREQUEST, $this->header);    

			curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);     

			$this->Content = curl_exec($curl);        

			curl_close($curl);  

		}else{	

			fwrite($this->Fp, $this->header);

			while (!feof($this->Fp))

			{

				$this->Content .= fgets($this->Fp, 128);

			}

			fclose($this->Fp);

		}

		preg_match("/Content-Length:.?(\d+)/", $this->Content, $matches);

		$length = $matches[1];

		$this->Content = substr($this->Content, - $length);

		return $this->Content;

	}

}

?>

然后，调用接口，最后解析Json就行了

<?php

require_once('QQClass.class.php');

$url = 'http://v.t.qq.com/cgi-bin/weiboshow?f=p&tweetflag=1&fansflag=0&fansnum=0&name=bbayou&sign=7bf0ab71321a3f376b710d2c739a2a8788ee1b0f';

$QQ = new QQClass($url);

$QQdata = $QQ->Get_json();

//ToDo json_decode();

?>

http://www.auu.name/604/index.html