How to deploy a nutch job on google dataproc? -


i have been trying deploy nutch job (with custom plugins) on google hadoop dataproc cluster i've been encountering many errors (some basic suspect).

i need step step explicit guide on how this. guide should include how set permissions , access file both in gs bucket , on local file system (windows 7).

i have tried configuration no success:

region: global cluster: first-cluster job type: hadoop jar files gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/depl‌​oy/apache-nutch-1.12‌​-snapshot.job main class or jar: gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-<p>asia/ deploy/bin/nutch arguments: gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/seed‌​.txt, -depth 4

i have tried:

region: global  cluster: first-cluster job type: hadoop jar files gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/depl‌​oy/apache-nutch-1.12‌​-snapshot.job main class or jar: gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/ deploy/bin/crawl arguments: gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/seed‌​.txt, -depth 4

and:

region: global cluster: first-cluster job type: hadoop jar files gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/depl‌​oy/apache-nutch-1.12‌​-snapshot.job main class or jar: org.apache.nutch.crawl.crawl arguments: gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/seed‌​.txt, -depth 4

follow up: have made progress getting error now:

 17/07/28 18:59:11 info crawl.injector: injector: starting @ 2017-07-28 18:59:11 17/07/28 18:59:11 info crawl.injector: injector: crawldb: https://console.cloud.google.com/storage/browser/dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/urls 17/07/28 18:59:11 info crawl.injector: injector: urldir: https://console.cloud.google.com/storage/browser/dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/crawldb 17/07/28 18:59:11 info configuration.deprecation: mapred.temp.dir deprecated. instead, use mapreduce.cluster.temp.dir 17/07/28 18:59:11 info crawl.injector: injector: converting injected urls crawl db entries. 17/07/28 18:59:11 info gcs.googlehadoopfilesystembase: ghfs version: 1.6.1-hadoop2 17/07/28 18:59:11 error crawl.injector: injector: java.lang.illegalargumentexception: wrong fs: https://console.cloud.google.com/storage/browser/dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/urls, expected: hdfs://first-cluster-m     @ org.apache.hadoop.fs.filesystem.checkpath(filesystem.java:648)     @ org.apache.hadoop.hdfs.distributedfilesystem.getpathname(distributedfilesystem.java:194)     @ org.apache.hadoop.hdfs.distributedfilesystem.access$000(distributedfilesystem.java:106)     @ org.apache.hadoop.hdfs.distributedfilesystem$22.docall(distributedfilesystem.java:1305)     @ org.apache.hadoop.hdfs.distributedfilesystem$22.docall(distributedfilesystem.java:1301)     @ org.apache.hadoop.fs.filesystemlinkresolver.resolve(filesystemlinkresolver.java:81)     @ org.apache.hadoop.hdfs.distributedfilesystem.getfilestatus(distributedfilesystem.java:1301)     @ org.apache.hadoop.fs.filesystem.exists(filesystem.java:1426)     @ org.apache.nutch.crawl.injector.inject(injector.java:298)     @ org.apache.nutch.crawl.injector.run(injector.java:379)     @ org.apache.hadoop.util.toolrunner.run(toolrunner.java:70)     @ org.apache.nutch.crawl.injector.main(injector.java:369)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:62)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:498)     @ com.google.cloud.hadoop.services.agent.job.shim.hadooprunclassshim.main(hadooprunclassshim.java:19)

i know has filesystem. how access gcp filesystem , hadoop file?

follow up: have made progress config:

{ "reference": { "projectid": "ageless-valor-174413", "jobid": "108a7d43-671a-4f61-8ba8-b87010a8a823" }, "placement": { "clustername": "first-cluster", "clusteruuid": "f3795563-bd44-4896-bec7-0eb81a3f685a" }, "status": { "state": "error", "details": "google cloud dataproc agent reports job failure. if logs available, can found in 'gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/google-cloud-dataproc-metainfo/f3795563-bd44-4896-bec7-0eb81a3f685a/jobs/108a7d43-671a-4f61-8ba8-b87010a8a823/driveroutput'.", "statestarttime": "2017-07-28t18:59:13.518z" }, "statushistory": [ { "state": "pending", "statestarttime": "2017-07-28t18:58:57.660z" }, { "state": "setup_done", "statestarttime": "2017-07-28t18:59:00.811z" }, { "state": "running", "statestarttime": "2017-07-28t18:59:02.347z" } ], "driveroutputresourceuri": "gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/google-cloud-dataproc-metainfo/f3795563-bd44-4896-bec7-0eb81a3f685a/jobs/108a7d43-671a-4f61-8ba8-b87010a8a823/driveroutput", "drivercontrolfilesuri": "gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/google-cloud-dataproc-metainfo/f3795563-bd44-4896-bec7-0eb81a3f685a/jobs/108a7d43-671a-4f61-8ba8-b87010a8a823/", "hadoopjob": { "mainclass": "org.apache.nutch.crawl.injector", "args": [ "https://console.cloud.google.com/storage/browser/dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/urls/", "https://console.cloud.google.com/storage/browser/dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/crawldb/" ], "jarfileuris": [ "gs://dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/apache-nutch-1.12-snapshot.job" ], "loggingconfig": {} } }

but getting error now:

 17/07/28 18:59:11 info crawl.injector: injector: starting @ 2017-07-28 18:59:11 17/07/28 18:59:11 info crawl.injector: injector: crawldb: https://console.cloud.google.com/storage/browser/dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/urls 17/07/28 18:59:11 info crawl.injector: injector: urldir: https://console.cloud.google.com/storage/browser/dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/crawldb 17/07/28 18:59:11 info configuration.deprecation: mapred.temp.dir deprecated. instead, use mapreduce.cluster.temp.dir 17/07/28 18:59:11 info crawl.injector: injector: converting injected urls crawl db entries. 17/07/28 18:59:11 info gcs.googlehadoopfilesystembase: ghfs version: 1.6.1-hadoop2 17/07/28 18:59:11 error crawl.injector: injector: java.lang.illegalargumentexception: wrong fs: https://console.cloud.google.com/storage/browser/dataproc-60f583ce-a087-42b1-a62e-4319584631d3-asia/deploy/urls, expected: hdfs://first-cluster-m     @ org.apache.hadoop.fs.filesystem.checkpath(filesystem.java:648)     @ org.apache.hadoop.hdfs.distributedfilesystem.getpathname(distributedfilesystem.java:194)     @ org.apache.hadoop.hdfs.distributedfilesystem.access$000(distributedfilesystem.java:106)     @ org.apache.hadoop.hdfs.distributedfilesystem$22.docall(distributedfilesystem.java:1305)     @ org.apache.hadoop.hdfs.distributedfilesystem$22.docall(distributedfilesystem.java:1301)     @ org.apache.hadoop.fs.filesystemlinkresolver.resolve(filesystemlinkresolver.java:81)     @ org.apache.hadoop.hdfs.distributedfilesystem.getfilestatus(distributedfilesystem.java:1301)     @ org.apache.hadoop.fs.filesystem.exists(filesystem.java:1426)     @ org.apache.nutch.crawl.injector.inject(injector.java:298)     @ org.apache.nutch.crawl.injector.run(injector.java:379)     @ org.apache.hadoop.util.toolrunner.run(toolrunner.java:70)     @ org.apache.nutch.crawl.injector.main(injector.java:369)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:62)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:498)     @ com.google.cloud.hadoop.services.agent.job.shim.hadooprunclassshim.main(hadooprunclassshim.java:19)


Comments

Post a Comment

Popular posts from this blog

node.js - Node js - Trying to send POST request, but it is not loading javascript content -

i have been trying send post request website, , has not been loading full site. i guessing problem node js not loading javascript content, executing it. can see javascript in response, though never loads content javascript files producing in html. what mean is, doesn't execute javascript. how make node js code overcome this, , wait the full website load? have tried native code, , "request' module, both producing same errors. if helps, don't need native javascript. modules fine. some code have tried (request module): var options = { method: 'post', uri: 'https://url', formdata: { 'email': 'email', 'prize': '0', 'transactional': 'on' }, headers: { /* 'content-type': 'application/x-www-form-urlencoded' */ // set automatically } }; rp(options) .then(function (body) { console.log(body) }) .catch(function (err) { console.log(err) }); when fetch content...
Read more

javascript - Replicate keyboard event with html button -

i have tried few ways make happen. have javascript pinball game uses keyboard keys controls. converting application touch screen display. workaround, hoping overlay buttons when clicked simulate keyboard strokes. have done research , sounds should work. using jquery v3.2.1 here things have tried far... html element: <input type="button" class="coil"></input> $('.coil').trigger({type: 'keydown', which: 40, keycode: 40}); with this, not errors, nothing happens. var coilkey = $('body').keydown(function(e) { switch (e.keycode) { case 40: break;} }) $('.coil').click(coilkey); this 1 returns error in console... ((jquery.event.special[handleobj.origtype] || (intermediate value)).handle || handleobj.handler).apply not function below javascript game engine handles key listeners... addkeylistener: function (keyandlistener) { this.keylisteners.push(keyandlistener); }, // gi...
Read more

javascript - Web audio api 5.1 surround example not working in firefox -

i've been playing lott lately web audio api on both firefox , chrome. few days ago started experiment surround set. when trying surround audio in web audio api came notice example works fine in chrome v59 not in firefox v54. // create web audio api context var audioctx = new (window.audiocontext || window.webkitaudiocontext)(); audioctx.destination.channelinterpretation = 'discrete'; audioctx.destination.channelcountmode = 'explicit'; audioctx.destination.channelcount = 6; var oscillators = []; var merger = audioctx.createchannelmerger(6); console.log(audioctx.destination, merger); //merger.channelinterpretation = 'discrete'; //merger.channelcountmode = 'explicit'; //merger.channelcount = 6; var addoscilator = function(channel, frequency) { var oscillator = audioctx.createoscillator(); oscillator.frequency.value = frequency; // value in hertz oscillator.connect(merger,0,channel); oscillator.start(); oscillators.push(o...
Read more