scala - UDF in Spark being called twice -


i running problem udf being called twice. first sql command tried running is:

sqlcontext.sql("select * ( select city, ctof(avglow) feren citytemps ) x feren > 30.0 order feren desc").show();

where ctof udf.

i noticed ctof being called twice here (i suspect in clause , again in select clause). added print statements inside ctof function , saw ferenheit calculation above 30 degrees , ctof called twice. i'd rather called once since have extremely large dataframe.

after didn't have luck sql command, tried following:

val ferendf = sqlcontext.sql("select city, ctof(avglow) feren citytemps"); ferendf.registertemptable("ferentemp"); sqlcontext.sql("select city, feren ferentemp feren > 30.0 order feren desc").show();

my udf, ctof, still being called twice. there anyway can run these sql commands ctof being called once? fyi, using spark 2.1.0.


Comments

Post a Comment

Popular posts from this blog

Ansible warning on jinja2 braces on when -

i using code in vars my_var: "{{lookup('env','my_var') | default(true, true)}}" i using like - include: task.yml when: my_var this worked without issue in ansible 2.2 in ansible 2.3 warning this [warning]: when statements should not include jinja2 templating delimiters such {{ }} or {% %}. found: {{lookup('env','my_var') | default(true, true)}} so if use my_var: "lookup('env','my_var') | default(true, true)" , mean remove {{ }} , no warning my_var evaluated true no matter if pass false env variable. works fine in previous code gave how can fix it? the cleanest way set fact instead of using template in variable (it prevent lookup plugin being called multiple times in included tasks - not huge performance gain, cleaner): - set_fact: my_var: "{{lookup('env','my_var') | default(true, true)}}" also mind evaluate true in cases when environment my_var diff...
Read more

Parsing a protocol message from Go by Java -

i write server(go)-client(java) programe, , use protobuf communication. define proto file , share between server , client. in server side: compile shared proto file go protoc serialize object proto.marshal send client make request service in client side: compile shared proto file java protoc get bytes transferred via http deserialize bytes object. here following error: "com.google.protobuf.invalidprotocolbufferexception: while parsing protocol message, input ended unexpectedly in middle of field. mean either input has been truncated or embedded message misreported own length." i confirmed http work fine , value of bytes received in client side same bytes send server. have same problem on this? here proto file syntax = "proto3"; package tutorial; message person { string name = 1; int32 id = 2; string email = 3; enum phonetype { mobile = 0; home = 1; work = 2; } message phonenumber { ...
Read more

javascript - Replicate keyboard event with html button -

i have tried few ways make happen. have javascript pinball game uses keyboard keys controls. converting application touch screen display. workaround, hoping overlay buttons when clicked simulate keyboard strokes. have done research , sounds should work. using jquery v3.2.1 here things have tried far... html element: <input type="button" class="coil"></input> $('.coil').trigger({type: 'keydown', which: 40, keycode: 40}); with this, not errors, nothing happens. var coilkey = $('body').keydown(function(e) { switch (e.keycode) { case 40: break;} }) $('.coil').click(coilkey); this 1 returns error in console... ((jquery.event.special[handleobj.origtype] || (intermediate value)).handle || handleobj.handler).apply not function below javascript game engine handles key listeners... addkeylistener: function (keyandlistener) { this.keylisteners.push(keyandlistener); }, // gi...
Read more