【Python】Python-pool的几个例子

收集了几个python种进程池pool的使用例子，改写为py3版本。

1. pool.apply_async

进程非阻塞执行，输入不确定情况下用

默认情况下，Pool会创建固定数目的工作进程，并向这些工作进程传递作业，直到再没有更多作业为止。

当有新的请求提交到pool中时，如果池还没有满，那么就会创建一个新的进程用来执行该请求；

但如果池中的进程数已经达到规定最大值，那么该请求就会等待，直到池中有进程结束，才会创建新的进程来它。

#coding: utf-8
import multiprocessing
import time


def func(msg):
    print("msg:", msg)
    time.sleep(3)
    print("end")


if __name__ == "__main__":
    cores = multiprocessing.cpu_count()
    pool = multiprocessing.Pool(processes=cores)
    print("Adding tasks...")
    for i in range(cores):
        msg = "hello %d" %(i)
        pool.apply_async(func, (msg, ))   #维持执行的进程总数为processes，当一个进程执行完毕后会添加新的进程进去
    print("Starting tasks...")
    pool.close()
    pool.join()   #调用join之前，先调用close函数，否则会出错。执行完close后不会有新的进程加入到pool,join函数等待所有子进程结束
    print("Sub-process(es) done.")

测试结果：本人用spyder测试，子进程print无输出结果。有网友说子进程的stdio需要通过变量接受，而不是那样直接打印。

于是，我首先改了一下代码，获取返回值，证明进程已经执行了：

#coding: utf-8
import multiprocessing
import time

def func(msg):
    print("msg:", msg)
    time.sleep(3)
    print("end")
    return msg

if __name__ == "__main__":
    cores = multiprocessing.cpu_count()
    pool = multiprocessing.Pool(processes=cores)
    print("Adding tasks...")
    results = []
    for i in range(cores):
        msg = "hello %d" %(i)
        results.append(pool.apply_async(func, (msg, )))   #维持执行的进程总数为processes，当一个进程执行完毕后会添加新的进程进去
    print("Starting tasks...")
    pool.close()
    pool.join()   #调用join之前，先调用close函数，否则会出错。执行完close后不会有新的进程加入到pool,join函数等待所有子进程结束
    print([r.get() for r in results])
    print("Sub-process(es) done.")

怎么让print内容打印出来呢。Stack over flow上说明这是使用的IDE的问题，无法打印子进程结果：

This is an issue with IDLE, which you're using to run your code. IDLE does a fairly basic emulation of a terminal for handling the output of a program you run in it. It cannot handle subprocesses though, so while they'll run just fine in the background, you'll never see their output.

The simplest fix is to simply run your code from the command line.

An alternative might be to use a more sophisticated IDE. There are a bunch of them listed on the Python wiki, though I'm not sure which ones have better terminal emulation for multiprocessing output.

2. process

当被操作对象数目不大时，可以直接利用multiprocessing中的Process动态成生多个进程。

但如果任务多，需要用pool或者队列来控制。

#coding=utf-8
import multiprocessing

def do(n) :
  #获取当前线程的名字
  name = multiprocessing.current_process().name
  print(name,'starting')
  print("worker ", n)
  return 

if __name__ == '__main__' :
  numList = []
  for i in range(4) :
    p = multiprocessing.Process(target=do, args=(i,))
    numList.append(p)
    p.start()
    p.join()
    print("Process end.")