python - How to dynamically iterate over the output of an upstream task to create parallel tasks in airflow? -


consider following example of dag first task, get_id_creds, extracts list of credentials database. operation tells me users in database able run further data preprocessing on , writes ids file /tmp/ids.txt. scan ids dag , use them generate list of upload_transaction tasks can run in parallel.

my question is: there more idiomatically correct, dynamic way using airflow? have here feels clumsy , brittle. how can directly pass list of valid ids 1 process defines subsequent downstream processes? 

from datetime import datetime, timedelta import os import sys  airflow.models import dag airflow.operators.python_operator import pythonoperator  import ds_dependencies  script_path = os.getenv('dash_preproc_path') if script_path:     sys.path.insert(0, script_path)     import dash_workers else:     print('define dash_preproc_path value in environmental variables')     sys.exit(1)  default_args = {   'start_date': datetime.now(),   'schedule_interval': none }  dag = dag(   dag_id='dash_preproc',   default_args=default_args )  get_id_creds = pythonoperator(     task_id='get_id_creds',     python_callable=dash_workers.get_id_creds,      provide_context=true,     dag=dag)  open('/tmp/ids.txt', 'r') infile:     ids = infile.read().splitlines()  uid in uids:     upload_transactions = pythonoperator(         task_id=uid,         python_callable=dash_workers.upload_transactions,         op_args=[uid],         dag=dag)     upload_transactions.set_downstream(get_id_creds)

considering apache airflow workflow management tool, ie. determines dependencies between task user defines in comparison (as example) apache nifi dataflow management tool, ie. dependencies here data transferd through tasks.

that said, think approach quit right (my comment based on code posted) but airflow offers concept called xcom. allows tasks "cross-communicate" between them passing data. how big should passed data ? test! should not big. think in form of key,value pairs , stored in airflow meta-database,ie can't pass files example list ids work.

like said should test self. happy know experience. here example dag demonstrates use of xcom , here necessary documentation. cheers!


Comments

Post a Comment

Popular posts from this blog

node.js - Node js - Trying to send POST request, but it is not loading javascript content -

i have been trying send post request website, , has not been loading full site. i guessing problem node js not loading javascript content, executing it. can see javascript in response, though never loads content javascript files producing in html. what mean is, doesn't execute javascript. how make node js code overcome this, , wait the full website load? have tried native code, , "request' module, both producing same errors. if helps, don't need native javascript. modules fine. some code have tried (request module): var options = { method: 'post', uri: 'https://url', formdata: { 'email': 'email', 'prize': '0', 'transactional': 'on' }, headers: { /* 'content-type': 'application/x-www-form-urlencoded' */ // set automatically } }; rp(options) .then(function (body) { console.log(body) }) .catch(function (err) { console.log(err) }); when fetch content...
Read more

javascript - Replicate keyboard event with html button -

i have tried few ways make happen. have javascript pinball game uses keyboard keys controls. converting application touch screen display. workaround, hoping overlay buttons when clicked simulate keyboard strokes. have done research , sounds should work. using jquery v3.2.1 here things have tried far... html element: <input type="button" class="coil"></input> $('.coil').trigger({type: 'keydown', which: 40, keycode: 40}); with this, not errors, nothing happens. var coilkey = $('body').keydown(function(e) { switch (e.keycode) { case 40: break;} }) $('.coil').click(coilkey); this 1 returns error in console... ((jquery.event.special[handleobj.origtype] || (intermediate value)).handle || handleobj.handler).apply not function below javascript game engine handles key listeners... addkeylistener: function (keyandlistener) { this.keylisteners.push(keyandlistener); }, // gi...
Read more

javascript - Web audio api 5.1 surround example not working in firefox -

i've been playing lott lately web audio api on both firefox , chrome. few days ago started experiment surround set. when trying surround audio in web audio api came notice example works fine in chrome v59 not in firefox v54. // create web audio api context var audioctx = new (window.audiocontext || window.webkitaudiocontext)(); audioctx.destination.channelinterpretation = 'discrete'; audioctx.destination.channelcountmode = 'explicit'; audioctx.destination.channelcount = 6; var oscillators = []; var merger = audioctx.createchannelmerger(6); console.log(audioctx.destination, merger); //merger.channelinterpretation = 'discrete'; //merger.channelcountmode = 'explicit'; //merger.channelcount = 6; var addoscilator = function(channel, frequency) { var oscillator = audioctx.createoscillator(); oscillator.frequency.value = frequency; // value in hertz oscillator.connect(merger,0,channel); oscillator.start(); oscillators.push(o...
Read more