Restarting Kafka Connect S3 Sink Task Loses Position, Completely Rewrites everything -


after restarting kafka connect s3 sink task, restarted writing way beginning of topic , wrote duplicate copies of older records. in other words, kafka connect seemed lose place.

so, imagine kafka connect stores current offset position information in internal connect-offsets topic. topic empty presume part of problem.

the other 2 internal topics connect-statuses , connect-configs not empty. connect-statuses has 52 entries. connect-configs has 6 entries; 3 each of 2 sink connectors have configured: connector-<name>, task-<name>-0, commit-<name>.

i manually created internal kafka connect topics specified in docs before running this:

/usr/bin/kafka-topics --create --zookeeper localhost:2181 --topic connect-configs --replication-factor 3 --partitions 1 --config cleanup.policy=compact /usr/bin/kafka-topics --create --zookeeper localhost:2181 --topic connect-offsets --replication-factor 3 --partitions 50 --config cleanup.policy=compact /usr/bin/kafka-topics --create --zookeeper localhost:2181 --topic connect-statuses --replication-factor 3 --partitions 10 --config cleanup.policy=compact

i can verify connect-offsets topic seems created correctly:

/usr/bin/kafka-topics --zookeeper localhost:2181 --describe --topic connect-offsets topic:connect-offsets   partitioncount:50   replicationfactor:3 configs:cleanup.policy=compact     topic: connect-offsets  partition: 0    leader: 1   replicas: 1,2,3 isr: 1,2,3     topic: connect-offsets  partition: 1    leader: 2   replicas: 2,3,1 isr: 2,3,1     topic: connect-offsets  partition: 2    leader: 3   replicas: 3,1,2 isr: 3,1,2   <snip>

this 3 server cluster running confluent platform v3.2.1 running kafka 10.2.1.

is connect-offsets supposed empty? why else kafka connect restart @ beginning of topic when restarting task?

update: response randall hauch's answer.

  • explanation regarding source connector offsets vs sink connector offsets explains empty connect-offsets. explanation!
  • i'm not changing connector name.
  • if connector down ~five days , restarted afterwards, there reason connector offset position expire , reset? see __consumer_offsets has cleanup.policy=compact
  • auto.offset.reset should take affect if there no position in __consumer_offsets, right?

i'm using system defaults. sink config json follows. i'm using simple custom partitioner partition on avro datetime field rather wallclock time. feature seems have been added in confluent v3.2.2 won't need custom plugin functionality. i'm hoping skip confluent v3.2.2 , go straight v3.3.0 when available.

{   "name": "my-s3-sink",    "tasks.max": 1,   "topics": "my-topic",   "flush.size": 10000,    "connector.class": "io.confluent.connect.s3.s3sinkconnector",   "storage.class": "io.confluent.connect.s3.storage.s3storage",   "format.class": "io.confluent.connect.s3.format.avro.avroformat",   "schema.generator.class": "io.confluent.connect.storage.hive.schema.timebasedschemagenerator",   "partitioner.class": "mycompany.partitioner.timefieldpartitioner",    "s3.bucket.name": "my-bucket",   "s3.region": "us-west-2",    "partition.field.name": "timestamp",    "locale": "us",   "timezone": "utc",   "path.format": "'year'=yyyy/'month'=mm/'day'=dd/'hour'=hh",    "schema.compatibility": "none",    "key.converter": "io.confluent.connect.avro.avroconverter",   "key.converter.schema.registry.url": "http://localhost:8081",   "value.converter": "io.confluent.connect.avro.avroconverter",   "value.converter.schema.registry.url": "http://localhost:8081" }

the default offset retention period kafka consumers 24 hours (1440 minutes). if stop connector , therefore make no new commits longer 24 hours offsets expire , start on new consumer when restart. can modify retention period on __consumer_offsets topic using offsets.retention.minutes parameter


Comments

Post a Comment

Popular posts from this blog

node.js - Node js - Trying to send POST request, but it is not loading javascript content -

i have been trying send post request website, , has not been loading full site. i guessing problem node js not loading javascript content, executing it. can see javascript in response, though never loads content javascript files producing in html. what mean is, doesn't execute javascript. how make node js code overcome this, , wait the full website load? have tried native code, , "request' module, both producing same errors. if helps, don't need native javascript. modules fine. some code have tried (request module): var options = { method: 'post', uri: 'https://url', formdata: { 'email': 'email', 'prize': '0', 'transactional': 'on' }, headers: { /* 'content-type': 'application/x-www-form-urlencoded' */ // set automatically } }; rp(options) .then(function (body) { console.log(body) }) .catch(function (err) { console.log(err) }); when fetch content...
Read more

javascript - Replicate keyboard event with html button -

i have tried few ways make happen. have javascript pinball game uses keyboard keys controls. converting application touch screen display. workaround, hoping overlay buttons when clicked simulate keyboard strokes. have done research , sounds should work. using jquery v3.2.1 here things have tried far... html element: <input type="button" class="coil"></input> $('.coil').trigger({type: 'keydown', which: 40, keycode: 40}); with this, not errors, nothing happens. var coilkey = $('body').keydown(function(e) { switch (e.keycode) { case 40: break;} }) $('.coil').click(coilkey); this 1 returns error in console... ((jquery.event.special[handleobj.origtype] || (intermediate value)).handle || handleobj.handler).apply not function below javascript game engine handles key listeners... addkeylistener: function (keyandlistener) { this.keylisteners.push(keyandlistener); }, // gi...
Read more

javascript - Web audio api 5.1 surround example not working in firefox -

i've been playing lott lately web audio api on both firefox , chrome. few days ago started experiment surround set. when trying surround audio in web audio api came notice example works fine in chrome v59 not in firefox v54. // create web audio api context var audioctx = new (window.audiocontext || window.webkitaudiocontext)(); audioctx.destination.channelinterpretation = 'discrete'; audioctx.destination.channelcountmode = 'explicit'; audioctx.destination.channelcount = 6; var oscillators = []; var merger = audioctx.createchannelmerger(6); console.log(audioctx.destination, merger); //merger.channelinterpretation = 'discrete'; //merger.channelcountmode = 'explicit'; //merger.channelcount = 6; var addoscilator = function(channel, frequency) { var oscillator = audioctx.createoscillator(); oscillator.frequency.value = frequency; // value in hertz oscillator.connect(merger,0,channel); oscillator.start(); oscillators.push(o...
Read more