How to improve spark perf by joining dataframe on a known key -


i have 2 big parquet dataframes , want join them on userid.

what should high performance :

should modify code write files in order :

  • partitionby on userid (very sparse).
  • partitionby on first n char of userid (afaik, if data partitioned on same key, join occur no shuffle)

on read side, better use rdd or dataframe ?

you can perform bucketby operation before save parquet file.

val num_buckets = 20 df1.write.mode(savemode.overwrite).bucketby(num_buckets, "userid").saveastable("bucketed_large_table_1") df2.write.mode(savemode.overwrite).bucketby(num_buckets, "userid").saveastable("bucketed_large_table_2") spark.sql("select * join b on a.num1 = b.num2").collect()

doing way prevent shuffle when join operation performed.

keep in mind in order need enable hive .enablehivesupport, save parquet operation doesn't support bucketby method.


Comments

Post a Comment

Popular posts from this blog

python - Selenium remoteWebDriver (& SauceLabs) Firefox moseMoveTo action exception -

while performing simple hover test on firefox through selenium (3.4.0) python bindings (3.4.3) using of available os in saucelabs (except linux, latest available firefox version old), performing following actionchain: hover = actionchains(driver) hover.move_to_element(elm_men_menu).perform() it throws following error: taceback: file [...] in testhover hover.perform() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 80, in perform action() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 290, in <lambda> command.move_to, {'element': to_element.id})) file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute self.error_handler.check_response(response) file "/usr/local/lib/python2.7/...
Read more

html - How to custom Bootstrap grid height? -

why bootstrap grid has height 591.067 ? did height came ? .main-post { padding: 20px; background-color: #fff; border: 1px solid #d8d8d8; margin-bottom: 100px; } .main-post .post-categories { margin-bottom: 10px } .main-post h3 { margin: 0 0 10px; color: #777; letter-spacing: -1px; font-weight: bold } .main-post .post-author, .main-post .post-date, .main-post .post-comments { font-size: 12px } .main-post img { display: block; margin: 10px 0; } .main-post .post-summary { line-height: 1.7; color: #888 } .main-post { color: #999 } <div class="col-sm-6"> <div class="main-post"> <h3 class="post-title"> <a href="#"> post title </a> </...
Read more

Ansible warning on jinja2 braces on when -

i using code in vars my_var: "{{lookup('env','my_var') | default(true, true)}}" i using like - include: task.yml when: my_var this worked without issue in ansible 2.2 in ansible 2.3 warning this [warning]: when statements should not include jinja2 templating delimiters such {{ }} or {% %}. found: {{lookup('env','my_var') | default(true, true)}} so if use my_var: "lookup('env','my_var') | default(true, true)" , mean remove {{ }} , no warning my_var evaluated true no matter if pass false env variable. works fine in previous code gave how can fix it? the cleanest way set fact instead of using template in variable (it prevent lookup plugin being called multiple times in included tasks - not huge performance gain, cleaner): - set_fact: my_var: "{{lookup('env','my_var') | default(true, true)}}" also mind evaluate true in cases when environment my_var diff...
Read more