【机器学习】pytorch中Dataloader的坑

基于pytorch训练一个模型，数据集size大概七十万
训练起来很慢
疯狂找原因，后来经过一行行调试，发现是遍历Dataloader时太慢了
正在寻找解决办法。。(待更新2020.1.5)

排坑1 ：有大佬说PIL库比opencv慢，于是把代码里PIL库打开图片操作改成了opencv

img = Image.open(img_name)
#省略xxx行
t1 = time.time()
for o in train_loader:
    t2 = time.time()
    print(t2 - t1)
    break
‘t2 - t1 = 6.84’

↓
↓
↓

img = cv2.imread(img_name)
#省略xxx行
t1 = time.time()
for o in train_loader:
    t2 = time.time()
    print(t2 - t1)
    break
‘t2 - t1 = 7.56’

可以发现，单纯打开图片操作，opencv还不如PIL
(待更新2020.1.5)

找到问题了！
经过我多方调试，找到耗时元凶！

# @count_time
class FaceDataset():
    
    def __init__(self, data, transforms= None):
        '''
        :param lines: src_line
        :param transform: data transform
        '''
        self.data = data
        self.transforms = transforms
           
    def __len__(self):
        return len(self.data)
#     @count_time
    def __getitem__(self, idx):
#         t1 = time.time()
		img_name, img_label, img_offsets = parse_line(self.data.values[idx]) 
        img_name, img_label, img_offsets = parse_line(self.data.iloc[idx,:])
#         t2 = time.time()
        #打开图片
        img_name = img_name
#         t3 = time.time()
        img = Image.open(img_name)
#         t4 = time.time()
        img = np.asarray(img,dtype= 'uint8')
        img_offsets = np.asarray(img_offsets, dtype = 'float64')
        sample = {'image':img, 'label':img_label, 'offsets': img_offsets}
#         t5 = time.time()
        sample = self.transforms(sample)
#         t6 = time.time()
#         print(t2-t1,t3-t2,t4-t3,t5-t4,t6-t5)
        return sample

这步中：

#我一开始是这么写的，先把df数据转为numpy数据才索引了，数据集比较大的话这步需要耗时很久
img_name, img_label, img_offsets = parse_line(self.data.values[idx]) 
#改成直接datafram索引，快了许多倍
img_name, img_label, img_offsets = parse_line(self.data.iloc[idx,:])

问题阶段性解决
不过性能瓶颈还是在读数据上，gpu利用率很低，但是可以接受啦，先跑起来再说~