hadoop - Use Spark 2.2.0 to read from Hive metastore 2.x -


this question has answer here:

prior version 2.2.0 spark unable communicate hive 2.x stuck using hive 1 + spark 1/2. have read in both:

https://issues.apache.org/jira/browse/spark-18112 https://spark.apache.org/releases/spark-release-2-2-0.html

it possible use spark 2 + hive 2 i'm still facing issues. using pre-compiled spark-without-hadoop, following error when accessing temporal hive table:

exception in thread "main" java.lang.illegalargumentexception: unable instantiate sparksession hive support because hive classes not found. @ org.apache.spark.sql.sparksession$builder.enablehivesupport(sparksession.scala:845) @ io.bigdatabenchmark.v2.queries.q05.logisticregression$.main(logisticregression.scala:87) @ io.bigdatabenchmark.v2.queries.q05.logisticregression.main(logisticregression.scala) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:62) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:498) @ org.apache.spark.deploy.sparksubmit$.org$apache$spark$deploy$sparksubmit$$runmain(sparksubmit.scala:755) @ org.apache.spark.deploy.sparksubmit$.dorunmain$1(sparksubmit.scala:180) @ org.apache.spark.deploy.sparksubmit$.submit(sparksubmit.scala:205) @ org.apache.spark.deploy.sparksubmit$.main(sparksubmit.scala:119) @ org.apache.spark.deploy.sparksubmit.main(sparksubmit.scala)

i solve issue compiling own version of spark options "-phive -phive-thriftserver" default spark build hive 1.2.1 bindings seen in documentation.

so, seems spark 2.2.0 solves issue of spark 2 -> hive 2 binding can't find proper way compile can access metastore schema 2.x.

thanks help!

add following dependency maven project.

<dependency>         <groupid>org.apache.spark</groupid>         <artifactid>spark-hive_2.11</artifactid>         <version>2.2.0</version>         <scope>provided</scope> </dependency>

Comments

Post a Comment

Popular posts from this blog

python - Selenium remoteWebDriver (& SauceLabs) Firefox moseMoveTo action exception -

while performing simple hover test on firefox through selenium (3.4.0) python bindings (3.4.3) using of available os in saucelabs (except linux, latest available firefox version old), performing following actionchain: hover = actionchains(driver) hover.move_to_element(elm_men_menu).perform() it throws following error: taceback: file [...] in testhover hover.perform() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 80, in perform action() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 290, in <lambda> command.move_to, {'element': to_element.id})) file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute self.error_handler.check_response(response) file "/usr/local/lib/python2.7/...
Read more

html - How to custom Bootstrap grid height? -

why bootstrap grid has height 591.067 ? did height came ? .main-post { padding: 20px; background-color: #fff; border: 1px solid #d8d8d8; margin-bottom: 100px; } .main-post .post-categories { margin-bottom: 10px } .main-post h3 { margin: 0 0 10px; color: #777; letter-spacing: -1px; font-weight: bold } .main-post .post-author, .main-post .post-date, .main-post .post-comments { font-size: 12px } .main-post img { display: block; margin: 10px 0; } .main-post .post-summary { line-height: 1.7; color: #888 } .main-post { color: #999 } <div class="col-sm-6"> <div class="main-post"> <h3 class="post-title"> <a href="#"> post title </a> </...
Read more

transpose - Maple isnt executing function but prints function term -

im using maple im in worksheetmode tried using maple input , 2d input , want transpose matrix a: := `<|>`(`<,>`(1, .5, -2), `<,>`(.5, 9/4+b, 5+3*b), `<,>`(-2, 5+3*b, 18+9*b+4*a)); b:= transpose(a); when exectute sheet dont translated values, there same input. matrix looks same input matrix plus function term. you can see picture in following link: why arent functions executed? meanwhile b:=a^+ doing right way , transposed matrix. other functions return function body instead needed values... if using 2d input mode (the default) space you've got between transpose , bracketed (b) interpreted multiplcation. rid of such space. also, either load package @ start of document like, with(linearalgebra): before calling transpose command package, or call it's full name like, linearalgebra:-transpose(b);
Read more