pdf - Document contains at least one immense term - Solr indexing error -


i facing issue 1 of pdf files presume fails indexed solr due large file size. have seen replies online advising change field type of 'content' 'text_general', have been using while particular pdf still cannot indexed.

error produced:

exception writing document id abc.com/files/hugepdf.pdf index; possible analysis error: document contains @ least 1 immense term in field="content" (whose utf8 encoding longer max length 32766), of skipped. please correct analyzer not produce such terms. prefix of first immense term is: '[66, 65, 82, 73, 78, 71, 32, 71, 76, 79, 66, 65, 76, 32, 79, 80, 80, 79, 82, 84, 85, 78, 73, 84, 73, 69, 83, 32, 85, 77]...', original message: bytes can @ 32766 in length; got 110482. perhaps document has indexed string field (solr.strfield) large

current schema of 'text_general'

  <fieldtype name="text_general" class="solr.textfield" positionincrementgap="100" multivalued="true">       <analyzer type="index">         <tokenizer class="solr.standardtokenizerfactory"/>         <filter class="solr.lowercasefilterfactory"/>         <filter class="solr.englishminimalstemfilterfactory"/>         <filter class="solr.truncatetokenfilterfactory" prefixlength="100"/>        </analyzer>       <analyzer type="query">         <tokenizer class="solr.standardtokenizerfactory"/>         <filter class="solr.lowercasefilterfactory"/>         <filter class="solr.englishminimalstemfilterfactory"/>       </analyzer>       <analyzer type="multiterm">         <tokenizer class="solr.standardtokenizerfactory"/>         <filter class="solr.lowercasefilterfactory"/>       </analyzer>   </fieldtype>

do note added 'truncatetokenfilterfactory' filter helped solve issues large pdf files. pdf exception.

questions

  1. what way make possible index such pdfs?
  2. on indexing failure, indexes not added solr (which wastes effort takes long time (a couple of hours) due 1 pdf file exceeds max size. there way around add successful indexes while rejecting specific indexes?

indexing pdf content known 'nightmare'. never 100% correct text extraction. suspect issue here extraction not working pdf, , returning huge pile of garbage. truncating not best approach, ignoring better. using 'text_general' not @ all.

some general guidelines be:

  1. do text extraction out of solr. yes handy use solr cell, real world pdf , volumes, worst case process hang (that worse dying). out of solr, in multiple threads, speed , make solr more reliable (less stress on it).
  2. use fallback library. probabaly usuing pdfbox (if using cell). if fails extract file, use second library (there several)

Comments

Post a Comment

Popular posts from this blog

node.js - Node js - Trying to send POST request, but it is not loading javascript content -

i have been trying send post request website, , has not been loading full site. i guessing problem node js not loading javascript content, executing it. can see javascript in response, though never loads content javascript files producing in html. what mean is, doesn't execute javascript. how make node js code overcome this, , wait the full website load? have tried native code, , "request' module, both producing same errors. if helps, don't need native javascript. modules fine. some code have tried (request module): var options = { method: 'post', uri: 'https://url', formdata: { 'email': 'email', 'prize': '0', 'transactional': 'on' }, headers: { /* 'content-type': 'application/x-www-form-urlencoded' */ // set automatically } }; rp(options) .then(function (body) { console.log(body) }) .catch(function (err) { console.log(err) }); when fetch content...
Read more

javascript - Replicate keyboard event with html button -

i have tried few ways make happen. have javascript pinball game uses keyboard keys controls. converting application touch screen display. workaround, hoping overlay buttons when clicked simulate keyboard strokes. have done research , sounds should work. using jquery v3.2.1 here things have tried far... html element: <input type="button" class="coil"></input> $('.coil').trigger({type: 'keydown', which: 40, keycode: 40}); with this, not errors, nothing happens. var coilkey = $('body').keydown(function(e) { switch (e.keycode) { case 40: break;} }) $('.coil').click(coilkey); this 1 returns error in console... ((jquery.event.special[handleobj.origtype] || (intermediate value)).handle || handleobj.handler).apply not function below javascript game engine handles key listeners... addkeylistener: function (keyandlistener) { this.keylisteners.push(keyandlistener); }, // gi...
Read more

javascript - Web audio api 5.1 surround example not working in firefox -

i've been playing lott lately web audio api on both firefox , chrome. few days ago started experiment surround set. when trying surround audio in web audio api came notice example works fine in chrome v59 not in firefox v54. // create web audio api context var audioctx = new (window.audiocontext || window.webkitaudiocontext)(); audioctx.destination.channelinterpretation = 'discrete'; audioctx.destination.channelcountmode = 'explicit'; audioctx.destination.channelcount = 6; var oscillators = []; var merger = audioctx.createchannelmerger(6); console.log(audioctx.destination, merger); //merger.channelinterpretation = 'discrete'; //merger.channelcountmode = 'explicit'; //merger.channelcount = 6; var addoscilator = function(channel, frequency) { var oscillator = audioctx.createoscillator(); oscillator.frequency.value = frequency; // value in hertz oscillator.connect(merger,0,channel); oscillator.start(); oscillators.push(o...
Read more