useing cURL to automate HTTP jobs

curl --cookie headers_and_cookies http://www.weibo.com
curl --data-urlencode "date=April 1" example.com/form.cgi

13. Debug
    13.1 Some debug tricks
    Many times when you run curl on a site, you'll notice that the site doesn't seem to respond the same way to your curl requests as it does to your browser's.
    Then you need to start making your curl requests more similar to your browser's requests:

• Use the --trace-ascii option to store fully detailed logs of the requests for easier analyzing and better understanding
• Make sure you check for and use cookies when needed (both reading with --cookie and writing with --cookie-jar)
• Set user-agent to one like a recent popular browser does
• Set referer like it is set by the browser
• If you use POST, make sure you send all the fields and in the same order as the browser does it.
  A very good helper to make sure you do this right, is the LiveHTTPHeader tool that lets you view all headers you send and receive with Mozilla/Firefox (even when using HTTPS). Chrome features similar functionality out of the box among the developer's tools.
  A more raw approach is to capture the HTTP traffic on the network with tools such as ethereal or tcpdump and check what headers that were sent and received by the browser. (HTTPS makes this technique inefficient.)