python - Empty dataframe falsely generated by groupby(pd.TimeGrouper('time_interval')).idxmin() -


i facing task of finding exact time @ measured value within time series minimal within equidistant time intervals of time series.

i tried use df.groupby(pd.timegrouper('time_interval')).idxmin() perform task, encounter unexpected (maybe wrong) behaviour of method: when using df.groupby(pd.timegrouper('time_interval')).idxmin() method on dataframe datetime indices contains (at least) 1 interval between 2 rows larger resample interval, generates empty dataframe instead of filling additional intervals 'nat' (like df.groupby(pd.timegrouper('time_interval')).xmin() fills additional intervals 'nan'). knows workaround problem (or there maybe bugfix method)? put minimal working example inline discussion @ end of post.

cheers,

simon

python version: python 3.6.0 :: anaconda 4.3.1 (64-bit)

pandas version: 0.19.2

import datetime import pandas pd  timestamp_list = [1493992554.897, 1493999093.997, 1493999108.733, 1493999116.101, 1493999117.943, 1493999119.785, 1493999121.627, 1493999123.469, 1493999125.311, 1493999127.153, 1493999128.995, 1493999130.837, 1493999132.679, 1493999134.521, 1493999136.363, 1493999138.205, 1493999140.047, 1493999141.889, 1493999143.731, 1493999145.573, 1493999147.415, 1493999149.257, 1493999151.099, 1493999152.941, 1493999154.783, 1493999156.625, 1493999158.467, 1493999160.309, 1493999162.151, 1493999163.993] value_list = [2.52962e-41, 2.52962e-41, 11.9625, 12.033420000000001, 12.069, 12.0784, 12.080933333333334, 12.080549999999999, 12.080233333333332, 12.078975, 12.033750000000001, 11.9472, 11.910966666666667, 11.902700000000001, 11.899766666666666, 11.898925, 11.898733333333332, 11.8987, 11.921174999999998, 11.982775, 12.010975000000002, 12.019466666666666, 12.021700000000001, 12.0224, 12.0225, 12.0226, 11.95525, 11.776133333333334, 11.65815, 11.624400000000001]  dt_list = [datetime.datetime.fromtimestamp(x) x in timestamp_list]  time_frame =  pd.dataframe(index=dt_list, data=value_list) time_frame.columns = ['value']  time_frame.head() # out[11]: #                                value # 2017-05-05 15:55:54.897  2.529620e-41  <- large time diff (larger resample length) # 2017-05-05 17:44:53.997  2.529620e-41  <- # 2017-05-05 17:45:08.733  1.196250e+01 # 2017-05-05 17:45:16.101  1.203342e+01 # 2017-05-05 17:45:17.943  1.206900e+01  # want resample dataframe , determine min in each interval # works fine:  tf_resampled_min = time_frame.groupby(pd.timegrouper('60000l')).min() tf_resampled_min.head()  #out[13]:     #                        value #2017-05-05 15:55:00  2.529620e-41 #2017-05-05 15:56:00           nan #2017-05-05 15:57:00           nan #2017-05-05 15:58:00           nan #2017-05-05 15:59:00           nan  # want determine exact time mmin occured, , here encounter problem:  tf_resampled_idxmin = time_frame.groupby(pd.timegrouper('60000l')).idxmin() tf_resampled_idxmin.head()  #out[14]: #empty dataframe #columns: [] #index: []  # expected like: #                         #2017-05-05 15:55:00  2017-05-05 15:55:54.897 #2017-05-05 15:56:00           nat #2017-05-05 15:57:00           nat #2017-05-05 15:58:00           nat #2017-05-05 15:59:00           nat  # output still able determine minidx in valid regions, empty dataframe, information lost.  # problem indeed time gap between first 2 entries. if remove them, get:  timestamp_list2 = [1493999093.997, 1493999108.733, 1493999116.101, 1493999117.943, 1493999119.785, 1493999121.627, 1493999123.469, 1493999125.311, 1493999127.153, 1493999128.995, 1493999130.837, 1493999132.679, 1493999134.521, 1493999136.363, 1493999138.205, 1493999140.047, 1493999141.889, 1493999143.731, 1493999145.573, 1493999147.415, 1493999149.257, 1493999151.099, 1493999152.941, 1493999154.783, 1493999156.625, 1493999158.467, 1493999160.309, 1493999162.151, 1493999163.993] value_list2 = [2.52962e-41, 11.9625, 12.033420000000001, 12.069, 12.0784, 12.080933333333334, 12.080549999999999, 12.080233333333332, 12.078975, 12.033750000000001, 11.9472, 11.910966666666667, 11.902700000000001, 11.899766666666666, 11.898925, 11.898733333333332, 11.8987, 11.921174999999998, 11.982775, 12.010975000000002, 12.019466666666666, 12.021700000000001, 12.0224, 12.0225, 12.0226, 11.95525, 11.776133333333334, 11.65815, 11.624400000000001]  dt_list2 = [datetime.datetime.fromtimestamp(x) x in timestamp_list2] time_frame2 =  pd.dataframe(index=dt_list2, data=value_list2) time_frame2.columns = ['value']  tf_resampled_idxmin2 = time_frame2.groupby(pd.timegrouper('60000l')).idxmin() tf_resampled_idxmin2.head()  #out[20]: #                                      value #2017-05-05 17:44:00 2017-05-05 17:44:53.997 #2017-05-05 17:45:00 2017-05-05 17:45:41.889 #2017-05-05 17:46:00 2017-05-05 17:46:03.993

i found workaround problem:

import datetime import pandas pd import numpy np  timestamp_list = [1493992554.897, 1493999093.997, 1493999108.733, 1493999116.101, 1493999117.943, 1493999119.785, 1493999121.627, 1493999123.469, 1493999125.311, 1493999127.153, 1493999128.995, 1493999130.837, 1493999132.679, 1493999134.521, 1493999136.363, 1493999138.205, 1493999140.047, 1493999141.889, 1493999143.731, 1493999145.573, 1493999147.415, 1493999149.257, 1493999151.099, 1493999152.941, 1493999154.783, 1493999156.625, 1493999158.467, 1493999160.309, 1493999162.151, 1493999163.993] value_list = [2.52962e-41, 2.52962e-41, 11.9625, 12.033420000000001, 12.069, 12.0784, 12.080933333333334, 12.080549999999999, 12.080233333333332, 12.078975, 12.033750000000001, 11.9472, 11.910966666666667, 11.902700000000001, 11.899766666666666, 11.898925, 11.898733333333332, 11.8987, 11.921174999999998, 11.982775, 12.010975000000002, 12.019466666666666, 12.021700000000001, 12.0224, 12.0225, 12.0226, 11.95525, 11.776133333333334, 11.65815, 11.624400000000001]  dt_list = [datetime.datetime.fromtimestamp(x) x in timestamp_list]  time_frame =  pd.dataframe(index=dt_list, data=value_list) time_frame.columns = ['value']  tf_resampled_idxmin = time_frame.resample("60000l").agg([lambda x: np.argmin(x) if len(x) > 0 else np.datetime64('nat')]) print(tf_resampled_idxmin)  #                                  value #                               <lambda> #2017-05-05 15:55:00 2017-05-05 15:55:54.897 #2017-05-05 15:56:00                     nat #2017-05-05 16:23:00                     nat #2017-05-05 16:24:00                     nat #...                                     ... #2017-05-05 17:17:00                     nat #2017-05-05 17:18:00                     nat #2017-05-05 17:43:00                     nat #2017-05-05 17:44:00 2017-05-05 17:44:53.997 #2017-05-05 17:45:00 2017-05-05 17:45:41.889 #2017-05-05 17:46:00 2017-05-05 17:46:03.993

the trick implement own version of idxmin() .agg([np.argmin()]) , lambda function catch case of empty list.


Comments

Post a Comment

Popular posts from this blog

python - Selenium remoteWebDriver (& SauceLabs) Firefox moseMoveTo action exception -

while performing simple hover test on firefox through selenium (3.4.0) python bindings (3.4.3) using of available os in saucelabs (except linux, latest available firefox version old), performing following actionchain: hover = actionchains(driver) hover.move_to_element(elm_men_menu).perform() it throws following error: taceback: file [...] in testhover hover.perform() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 80, in perform action() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 290, in <lambda> command.move_to, {'element': to_element.id})) file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute self.error_handler.check_response(response) file "/usr/local/lib/python2.7/...
Read more

html - How to custom Bootstrap grid height? -

why bootstrap grid has height 591.067 ? did height came ? .main-post { padding: 20px; background-color: #fff; border: 1px solid #d8d8d8; margin-bottom: 100px; } .main-post .post-categories { margin-bottom: 10px } .main-post h3 { margin: 0 0 10px; color: #777; letter-spacing: -1px; font-weight: bold } .main-post .post-author, .main-post .post-date, .main-post .post-comments { font-size: 12px } .main-post img { display: block; margin: 10px 0; } .main-post .post-summary { line-height: 1.7; color: #888 } .main-post { color: #999 } <div class="col-sm-6"> <div class="main-post"> <h3 class="post-title"> <a href="#"> post title </a> </...
Read more

transpose - Maple isnt executing function but prints function term -

im using maple im in worksheetmode tried using maple input , 2d input , want transpose matrix a: := `<|>`(`<,>`(1, .5, -2), `<,>`(.5, 9/4+b, 5+3*b), `<,>`(-2, 5+3*b, 18+9*b+4*a)); b:= transpose(a); when exectute sheet dont translated values, there same input. matrix looks same input matrix plus function term. you can see picture in following link: why arent functions executed? meanwhile b:=a^+ doing right way , transposed matrix. other functions return function body instead needed values... if using 2d input mode (the default) space you've got between transpose , bracketed (b) interpreted multiplcation. rid of such space. also, either load package @ start of document like, with(linearalgebra): before calling transpose command package, or call it's full name like, linearalgebra:-transpose(b);
Read more