Python并发之多线程
多线程的使用
例子
例1
# coding: utf-8
import threading
import urllib2
import Queue
def task(q, url):
result = urllib2.urlopen(url)
q.put(result)
q = Queue.Queue()
urls = ['http://www.baidu.com', 'http://www.qq.com', 'http://www.sina.com']
threads = []
for url in urls:
t = threading.Thread(target=task, args=(q, url))
threads.append(t)
t.start()
for t in threads:
t.join()
while not q.empty():
s = q.get()
print s.code, s.url
例2
from multiprocessing.dummy import Pool as ThreadPool
import urllib2
def task(url):
result = urllib2.urlopen(url)
return result.code, result.url
urls = ['http://www.baidu.com', 'http://www.qq.com', 'http://www.sina.com']
pool = ThreadPool(4)
results = pool.map(task, urls)
print results例2跟例1思路上本质是一致的，只是在充分利用库函数multiprocessing.dummy中pool.map的特性，让多线程代码在写法上更优雅。
pool.map的优势
Map is a cool little function, and the key to easily injecting parallelism into your Python code. For those unfamiliar, map is something lifted from functional languages like Lisp. It is a function which maps another function over a sequence.
Map handles the iteration over the sequence for us, applies the function, and stores all of the results in a handy list at the end.
使用thread的场景
Proper use of threads in Python is invariably connected to I/O operations (since CPython doesn’t use multiple cores to run CPU-bound tasks anyway, the only reason for threading is not blocking the process while there’s a wait for some I/O).
Queues are almost invariably the best way to farm out work to threads and/or collect the work’s results, by the way, and they’re intrinsically threadsafe so they save you from worrying about locks, conditions, events, semaphores, and other inter-thread coordination/communication concepts.
多线程小结：
因为GIL的限制，CPython在处理I/O等待时，适合使用多线程。如果想要充分利用多核CPU，处理CPU密集型的任务，最好是使用多进程（multiprocessing）。
多进程vs多线程
在操作系统中，进程是分配资源的基本单位，线程是调度的基本单位。因此，多进程消耗的资源更多，线程更加轻量级；创建进程一般比线程慢。多进程拥有不同的存储区间，多线程共享的内存。一个进程的崩溃，不会影响其他进程；一个线程的崩溃，可能破坏其他同属一个进程的工作线程。这也导致，在进程间共享数据和通信变得更加困难；由于多线程共享内存，为了防止多个线程在同一时间写同一段内存，在python中GIL被引入来解决这一问题。
Multiprocessing
Pros
- Separate memory space
- Code is usually straightforward
- Takes advantage of multiple CPUs & cores
- Avoids GIL limitations for cPython
- Eliminates most needs for synchronization primitives unless if you use shared memory (instead, it’s more of a communication model for IPC)
- Child processes are interruptible/killable
- Python multiprocessing module includes useful abstractions with an interface much like threading.Thread
- A must with cPython for CPU-bound processing
Cons
- IPC a little more complicated with more overhead (communication model vs. shared memory/objects)
- Larger memory footprint
Threading
Pros
- Lightweight - low memory footprint
- Shared memory - makes access to state from another context easier
- Allows you to easily make responsive UIs
- cPython C extension modules that properly release the GIL will run in parallel
- Great option for I/O-bound applications
Cons
- cPython - subject to the GIL. In Python, because of GIL (Global Interpreter Lock) a single python process cannot run threads in parallel (utilize multiple cores).
- Not interruptible/killable
- If not following a command queue/message pump model (using the Queue module), then manual use of synchronization primitives become a necessity (decisions are needed for the granularity of locking)
- Code is usually harder to understand and to get right - the potential for race conditions increases dramatically
参考：
- GIL是什么鬼？参考：python最难的问题
- multiprocessing-vs-threading-python