tensorflow - ResourceExhaustedError during retraining in Object detection API on my own data set -


i getting resourceexhaustederror error while training object detection api in tensorflow. gpu 1080 ti (11 gb) . when starts training gpu takes memory . if train scratch runs perfectly. if try train on pretrained model (coco data set). trains few hundred steps , fails.

with error messege :

resource exhausted: ran out of gpu memory when allocating 0 bytes     [[node: loss/rpnloss/loss_1/softmaxcrossentropywithlogits = softmaxcrossentropywithlogits[t=dt_float,_device="/job:localhost/replica:0/task:0/gpu:0(loss/rpnloss/loss_1/reshape_2, loss/rpnloss/loss_1/reshape_3)]] 

i tried 2 models(resnet_101, inception_resnet_v2) fails in both. after searching set session_config.gpu_option.allow_growth = true getting same error.

i tried tricks mentioned in https://github.com/tensorflow/models/issues/1766 no luck.


Comments

Popular posts from this blog

python - Selenium remoteWebDriver (& SauceLabs) Firefox moseMoveTo action exception -

html - How to custom Bootstrap grid height? -

transpose - Maple isnt executing function but prints function term -