neural network - keras loss jumps to zero randomly at the start of a new epoch -
i'm training network has multiple losses , both creating , feeding data network using generator.
i've checked structure of data , looks fine , trains pretty expected majority of time, @ random epoch every time, the training loss every prediction jumps say
# end of epoch 3 loss: 2.8845 to
# beginning of epoch 4 loss: 1.1921e-07 i thought data, however, can tell data fine , it's more suspicious because happen @ random epoch (could because of random data point chosen during sgd?) persist throughout rest of training. in if @ epoch 3, training loss decreases 1.1921e-07 continue way in epoch 4, epoch 5, etc.
however, there times when reaches epoch 5 , hasn't done yet , might @ epoch 6 or 7.
is there viable reason outside of data cause this? happen few fudgy data points causes fast?
thanks
edit:
results:
300/300 [==============================] - 339s - loss: 3.2912 - loss_1: 1.8683 - loss_2: 9.1352 - loss_3: 5.9845 - val_loss: 1.1921e-07 - val_loss_1: 1.1921e-07 - val_loss_2: 1.1921e-07 - val_loss_3: 1.1921e-07 the next epochs after have trainig loss 1.1921e-07
not entirely sure how satisfactory answer findings seem show using multiple categorical_crossentropy loss's seems result in super unstable network? swapping out other loss functions fixes problem data remaining unchanged.
Comments
Post a Comment