tensorflow - ResourceExhaustedError during retraining in Object detection API on my own data set -
i getting resourceexhaustederror error while training object detection api in tensorflow. gpu 1080 ti (11 gb) . when starts training gpu takes memory . if train scratch runs perfectly. if try train on pretrained model (coco data set). trains few hundred steps , fails.
with error messege :
resource exhausted: ran out of gpu memory when allocating 0 bytes [[node: loss/rpnloss/loss_1/softmaxcrossentropywithlogits = softmaxcrossentropywithlogits[t=dt_float,_device="/job:localhost/replica:0/task:0/gpu:0(loss/rpnloss/loss_1/reshape_2, loss/rpnloss/loss_1/reshape_3)]] i tried 2 models(resnet_101, inception_resnet_v2) fails in both. after searching set session_config.gpu_option.allow_growth = true getting same error.
i tried tricks mentioned in https://github.com/tensorflow/models/issues/1766 no luck.
Comments
Post a Comment