amazon web services - Query S3 logs content using Athena or DynamoDB -


i have use case query request url s3 logs. amazon has introduced athena query s3 file contents. best option respect cost , performance?

  1. use athena query s3 files request urls
  2. store metadata of each file request url information in dynamodb table query

amazon dynamodb poor choice running queries on web logs.

dynamodb super-fast, if retrieving data based upon primary key ("query"). if running query against all data in table (eg find particular ip address in key not indexed), dynamodb need scan through rows in table, takes lot of time ("scan"). example, if table configured 100 reads per second , scanning 10000 rows, take 100 seconds (100 x 100 = 10000).

tip: not full-table scans in nosql database.

amazon athena ideal scanning log files! there no need pre-load data - run query against logs stored in amazon s3. use standard sql find data you're seeking. plus, pay data read disk. file format bit weird, you'll need correct create table statement.

see: using aws athena query s3 server access logs

another choice use amazon redshift, can gbs, tbs , pbs of data across billions of rows. if going run frequent queries against log data, redshift great. however, being standard sql database, need pre-load data redshift. unfortunately, amazon s3 log files not in csv format, need etl files suitable format. isn't worthwhile occasional, ad-hoc requests.

many people use amazon elasticsearch service scanning log files. again, file format needs special handling , pipeline load data needs work, result near-realtime interactive analysis of s3 log files.

see: using elk stack analyze s3 logs


Comments

Post a Comment

Popular posts from this blog

node.js - Node js - Trying to send POST request, but it is not loading javascript content -

i have been trying send post request website, , has not been loading full site. i guessing problem node js not loading javascript content, executing it. can see javascript in response, though never loads content javascript files producing in html. what mean is, doesn't execute javascript. how make node js code overcome this, , wait the full website load? have tried native code, , "request' module, both producing same errors. if helps, don't need native javascript. modules fine. some code have tried (request module): var options = { method: 'post', uri: 'https://url', formdata: { 'email': 'email', 'prize': '0', 'transactional': 'on' }, headers: { /* 'content-type': 'application/x-www-form-urlencoded' */ // set automatically } }; rp(options) .then(function (body) { console.log(body) }) .catch(function (err) { console.log(err) }); when fetch content...
Read more

javascript - Replicate keyboard event with html button -

i have tried few ways make happen. have javascript pinball game uses keyboard keys controls. converting application touch screen display. workaround, hoping overlay buttons when clicked simulate keyboard strokes. have done research , sounds should work. using jquery v3.2.1 here things have tried far... html element: <input type="button" class="coil"></input> $('.coil').trigger({type: 'keydown', which: 40, keycode: 40}); with this, not errors, nothing happens. var coilkey = $('body').keydown(function(e) { switch (e.keycode) { case 40: break;} }) $('.coil').click(coilkey); this 1 returns error in console... ((jquery.event.special[handleobj.origtype] || (intermediate value)).handle || handleobj.handler).apply not function below javascript game engine handles key listeners... addkeylistener: function (keyandlistener) { this.keylisteners.push(keyandlistener); }, // gi...
Read more

javascript - Web audio api 5.1 surround example not working in firefox -

i've been playing lott lately web audio api on both firefox , chrome. few days ago started experiment surround set. when trying surround audio in web audio api came notice example works fine in chrome v59 not in firefox v54. // create web audio api context var audioctx = new (window.audiocontext || window.webkitaudiocontext)(); audioctx.destination.channelinterpretation = 'discrete'; audioctx.destination.channelcountmode = 'explicit'; audioctx.destination.channelcount = 6; var oscillators = []; var merger = audioctx.createchannelmerger(6); console.log(audioctx.destination, merger); //merger.channelinterpretation = 'discrete'; //merger.channelcountmode = 'explicit'; //merger.channelcount = 6; var addoscilator = function(channel, frequency) { var oscillator = audioctx.createoscillator(); oscillator.frequency.value = frequency; // value in hertz oscillator.connect(merger,0,channel); oscillator.start(); oscillators.push(o...
Read more