多进程读取操作

README

多进程操作现在已经是日常task了，我对代码进行了简单封装，希望能给读者带来收获

code

def multiprocess_run(process, obj_list, notebook=False):
    if notebook is True:
        from tqdm import tqdm_notebook as tqdm
    else:
        from tqdm import tqdm
    from multiprocessing import Pool
    import multiprocessing
    import time
    cpus = multiprocessing.cpu_count()
    pool = Pool(processes=cpus)
    result_list = []
    tic = time.time()
    for result in tqdm(pool.imap(process, obj_list)):
        result_list.append(result)
        sys.stdout.flush()
    toc = time.time()
    print('time waste', toc - tic)
    return result_list

中间notebook变量是因为有时要在Notebook中运行，tqdm的显示方式会有区别