tensorflow GPU训练loss与val loss值差距过大问题

问题

最近在ubuntu gpu上训练模型，训练十轮，结果如下

epoch,loss,lr,val_loss
200,nan,0.001,nan
200,0.002468767808750272,0.001,44.29948425292969
201,0.007177405059337616,0.001,49.16984176635742
202,0.012423301115632057,0.001,49.30305862426758
203,0.019116541370749474,0.001,48.27520751953125
204,0.02645580656826496,0.0005,48.38237762451172
205,0.03023119457066059,0.0005,48.2923469543457
206,0.034110430628061295,0.0005,48.55632781982422
207,0.03898066654801369,0.00025,48.616432189941406
208,0.04163944348692894,0.00025,48.568756103515625
209,0.04249056056141853,0.00025,48.68966293334961

发现loss只有0.0几，而val loss达到了四十多，而我是在已经训练好的模型的基础上，使用新的数据集，继续训练的，差距这么的大，肯定不是模型过拟合，这明显是有异常的，但是我在CPU上训练，结果val和val loss都是-15左右，同样的代码和数据，为什么会出现这么大的差异呢。

解决方案

原因：是因为我用的tensorflow 2.2版本+cuda10.1，而GPU是3090，cuda版本和gpu不适配，需要升级cuda，因此我使用tensorflow 2.4+cuda 11.0后，训练就正常了，

解决方法是在github上看见的
here