hadoop - How to fix size limit error when performing actions on hive table in pyspark -


i have hive table 4 billion rows need load pyspark. when try actions such counting against table, following exception (followed taskkilled exceptions): 

py4jjavaerror: error occurred while calling o89.count. : org.apache.spark.sparkexception: job aborted due stage failure: task 6732 in stage 13.0 failed 4 times, recent failure: lost task 6732.3 in stage 13.0 (tid 30759, some_server.xx.net, executor 38): org.apache.hive.com.google.protobuf.invalidprotocolbufferexception: protocol mess age large.  may malicious.  use codedinputstream.setsizelimit() increase size limi t.

my version of hbase 1.1.2.2.6.1.0-129 , unable upgrade @ time.

is there way can around issue without upgrading, maybe modifying environment variable or config somewhere, or passing argument pyspark via command line?

i no.

based on following jiras increasing protobuf size seems require code change since these jiras resolved code patches using codedinputstream suggested exception.

  • hdfs-6102 lower default maximum items per directory fix pb fsimage loading
  • hdfs-10312 large block reports may fail decode @ namenode due 64 mb protobuf maximum length restriction.
  • hbase-14076 resultserialization , mutationserialization can throw invalidprotocolbufferexception when serializing cell larger 64mb
  • hive-11592 orc metadata section can exceed protobuf message size limit
  • spark-19109 orc metadata section can exceed protobuf message size limit

Comments

Post a Comment

Popular posts from this blog

node.js - Node js - Trying to send POST request, but it is not loading javascript content -

i have been trying send post request website, , has not been loading full site. i guessing problem node js not loading javascript content, executing it. can see javascript in response, though never loads content javascript files producing in html. what mean is, doesn't execute javascript. how make node js code overcome this, , wait the full website load? have tried native code, , "request' module, both producing same errors. if helps, don't need native javascript. modules fine. some code have tried (request module): var options = { method: 'post', uri: 'https://url', formdata: { 'email': 'email', 'prize': '0', 'transactional': 'on' }, headers: { /* 'content-type': 'application/x-www-form-urlencoded' */ // set automatically } }; rp(options) .then(function (body) { console.log(body) }) .catch(function (err) { console.log(err) }); when fetch content...
Read more

javascript - Replicate keyboard event with html button -

i have tried few ways make happen. have javascript pinball game uses keyboard keys controls. converting application touch screen display. workaround, hoping overlay buttons when clicked simulate keyboard strokes. have done research , sounds should work. using jquery v3.2.1 here things have tried far... html element: <input type="button" class="coil"></input> $('.coil').trigger({type: 'keydown', which: 40, keycode: 40}); with this, not errors, nothing happens. var coilkey = $('body').keydown(function(e) { switch (e.keycode) { case 40: break;} }) $('.coil').click(coilkey); this 1 returns error in console... ((jquery.event.special[handleobj.origtype] || (intermediate value)).handle || handleobj.handler).apply not function below javascript game engine handles key listeners... addkeylistener: function (keyandlistener) { this.keylisteners.push(keyandlistener); }, // gi...
Read more

javascript - Web audio api 5.1 surround example not working in firefox -

i've been playing lott lately web audio api on both firefox , chrome. few days ago started experiment surround set. when trying surround audio in web audio api came notice example works fine in chrome v59 not in firefox v54. // create web audio api context var audioctx = new (window.audiocontext || window.webkitaudiocontext)(); audioctx.destination.channelinterpretation = 'discrete'; audioctx.destination.channelcountmode = 'explicit'; audioctx.destination.channelcount = 6; var oscillators = []; var merger = audioctx.createchannelmerger(6); console.log(audioctx.destination, merger); //merger.channelinterpretation = 'discrete'; //merger.channelcountmode = 'explicit'; //merger.channelcount = 6; var addoscilator = function(channel, frequency) { var oscillator = audioctx.createoscillator(); oscillator.frequency.value = frequency; // value in hertz oscillator.connect(merger,0,channel); oscillator.start(); oscillators.push(o...
Read more