multithreading - Python explicit next() slower than a for loop in multithreaded producer-consumer paradigm -
suppose have following code runs 10 iterations , calls generator gen_next_img_batch
:
for _ in get_next_img_batch(train_data_paths, classes, batch_size): pass
now, suppose replace following, equivalent (to mind) iteration:
for in range(10): next(get_next_img_batch(train_data_paths, classes, batch_size))
the conundrum have first snippet takes 17 seconds, while second takes 42 seconds. moreover, every iteration of second snippet takes more time previous iteration. is, iteration i+1 of second snippet takes longer iteration i. iterations of first snippet take same time.
an important fact seems generator gen_next_img_batch
iterated on takes data python queue populated in background thread. more specifically, in background, separate thread loading data queue, , every call next()
yields object queue. when asynchronous behaviour omitted, both snippets take same time.
they not equivalent. specifically, first fragment calls get_next_img_batch()
once, while second calls function 10 times. equivalent fragment might be:
it = get_next_img_batch(train_data_paths, classes, batch_size)) in range(10): next(it)
Comments
Post a Comment