Kafka Streams - Explain the reason why KTable and its associated Store only get updated every 30 seconds -


i have simple ktable definition generates store:

ktable<string, jsonnode> table = kstreambuilder.<string, jsonnode>table(orders_topic, orders_store); table.print();

i publish messages orders_topic store isn't updated until every 30 seconds. log there message committing because 30000ms time has elapsed:

2017-07-25 23:53:15.465 debug 17540 --- [ streamthread-1] o.a.k.c.consumer.internals.fetcher       : sending fetch partitions [orders-0] broker exprf026.suministrador:9092 (id: 0 rack: null) 2017-07-25 23:53:15.567  info 17540 --- [ streamthread-1] o.a.k.s.p.internals.streamthread         : stream-thread [streamthread-1] committing tasks because commit interval 30000ms has elapsed 2017-07-25 23:53:15.567  info 17540 --- [ streamthread-1] o.a.k.s.p.internals.streamthread         : stream-thread [streamthread-1] committing task streamtask 0_0 2017-07-25 23:53:15.567 debug 17540 --- [ streamthread-1] o.a.k.s.processor.internals.streamtask   : task [0_0] committing state 2017-07-25 23:53:15.567 debug 17540 --- [ streamthread-1] o.a.k.s.p.i.processorstatemanager        : task [0_0] flushing stores registered in state manager f2b9ff2b-62c3-470e-8df1-066cd1e3d5ec {"uid":"string","productid":0,"orderid":"f2b9ff2b-62c3-470e-8df1-066cd1e3d5ec","name":"orderplaced","state":"pending_product_reservation"} [ktable-source-0000000001]: f2b9ff2b-62c3-470e-8df1-066cd1e3d5ec , ({"uid":"string","productid":0,"orderid":"f2b9ff2b-62c3-470e-8df1-066cd1e3d5ec","name":"orderplaced","state":"pending_product_reservation"}<-null) 2017-07-25 23:53:15.569 debug 17540 --- [ streamthread-1] o.a.k.s.state.internals.threadcache      : thread order-service-streams-16941f70-87b3-45f4-88de-309e4fd22748-streamthread-1 cache stats on flush: #puts=1, #gets=1, #evicts=0, #flushes=1 2017-07-25 23:53:15.576 debug 17540 --- [ streamthread-1] o.a.k.s.p.internals.recordcollectorimpl  : task [0_0] flushing producer

i found property controls commit.interval.ms:

props.put(streamsconfig.commit_interval_ms_config, 10);

why set 30000ms default (sounds long time) , implications of changing 10ms?

if instead of ktable work kstream...

kstream<string, jsonnode> kstream = kstreambuilder.stream(orders_topic); kstream.print();

...i can see messages right away, without having wait 30000ms, why difference?

it's related memory management in particular, ktable caches: http://docs.confluent.io/current/streams/developer-guide.html#memory-management

ktable updated time , if use "interactive queries" access underlying state store, can each update immediately. however, ktable cache buffers updates reduce downstream load , each time commit triggered, cache needs flushed downstream avoid data loss in case if failure. if cache size small, might see downstream records if key get's evicted cache.

about commit interval: in general, commit interval set relatively large value, reduce commit load on brokers.


Comments

Post a Comment

Popular posts from this blog

node.js - Node js - Trying to send POST request, but it is not loading javascript content -

i have been trying send post request website, , has not been loading full site. i guessing problem node js not loading javascript content, executing it. can see javascript in response, though never loads content javascript files producing in html. what mean is, doesn't execute javascript. how make node js code overcome this, , wait the full website load? have tried native code, , "request' module, both producing same errors. if helps, don't need native javascript. modules fine. some code have tried (request module): var options = { method: 'post', uri: 'https://url', formdata: { 'email': 'email', 'prize': '0', 'transactional': 'on' }, headers: { /* 'content-type': 'application/x-www-form-urlencoded' */ // set automatically } }; rp(options) .then(function (body) { console.log(body) }) .catch(function (err) { console.log(err) }); when fetch content...
Read more

javascript - Replicate keyboard event with html button -

i have tried few ways make happen. have javascript pinball game uses keyboard keys controls. converting application touch screen display. workaround, hoping overlay buttons when clicked simulate keyboard strokes. have done research , sounds should work. using jquery v3.2.1 here things have tried far... html element: <input type="button" class="coil"></input> $('.coil').trigger({type: 'keydown', which: 40, keycode: 40}); with this, not errors, nothing happens. var coilkey = $('body').keydown(function(e) { switch (e.keycode) { case 40: break;} }) $('.coil').click(coilkey); this 1 returns error in console... ((jquery.event.special[handleobj.origtype] || (intermediate value)).handle || handleobj.handler).apply not function below javascript game engine handles key listeners... addkeylistener: function (keyandlistener) { this.keylisteners.push(keyandlistener); }, // gi...
Read more

javascript - Web audio api 5.1 surround example not working in firefox -

i've been playing lott lately web audio api on both firefox , chrome. few days ago started experiment surround set. when trying surround audio in web audio api came notice example works fine in chrome v59 not in firefox v54. // create web audio api context var audioctx = new (window.audiocontext || window.webkitaudiocontext)(); audioctx.destination.channelinterpretation = 'discrete'; audioctx.destination.channelcountmode = 'explicit'; audioctx.destination.channelcount = 6; var oscillators = []; var merger = audioctx.createchannelmerger(6); console.log(audioctx.destination, merger); //merger.channelinterpretation = 'discrete'; //merger.channelcountmode = 'explicit'; //merger.channelcount = 6; var addoscilator = function(channel, frequency) { var oscillator = audioctx.createoscillator(); oscillator.frequency.value = frequency; // value in hertz oscillator.connect(merger,0,channel); oscillator.start(); oscillators.push(o...
Read more