Read and concat csv files from past 14 days in python pandas -
i need write script read csv files specific names past 14 days (every day in morning), when concat gives me little cube (in jupyter-notebook), , sign there nothing.
def get_local_file(pdate, hour, path='/data/'): """get date+hour processing file local drive :param pdate: str processing date :param hour: str processing hour :param path: str path file location :return: pandas df retrieved dataframe """ sdate = pdate + '-' + str(hour) p_file in os.listdir(path): if fnmatch.fnmatch(p_file, 'abc_*'+sdate+'*.csv'): return path+p_file def get_files(pdate, path='/data/'): hours = [time(i).strftime('%h') in range(24)] filelist=[] hour in hours: filelist.append(get_local_file(pdate, hour)) return filelist end_datetime = datetime.combine(date.today(), time(0, 0, 0)) proc_datetime = end_datetime - timedelta(days=14) while proc_datetime <= end_datetime: proc_datetime += timedelta(days=1) = get_files(str(proc_datetime.date()).replace('-', '_')) frame = pd.dataframe() list_ = [] file_ in a: if file_ != none: df = pd.read_csv(file_,index_col=none, header=0, delimiter=';') list_.append(df) frame = pd.concat(list_) i'm pretty sure possible make code while loop , below simpler, have no idea how it.
if want create single dataframe bunch of .csv files can way:
- initialize empty list before loop
- loop on files, every 1 read in dataframe , append list
- concatenate list single dataframe after loop
i did not check if handling of dates , filenames correct here relevant changes code regarding concatenation part:
end_datetime = datetime.combine(date.today(), time(0, 0, 0)) proc_datetime = end_datetime - timedelta(days=14) list_ = [] while proc_datetime <= end_datetime: proc_datetime += timedelta(days=1) = get_files(str(proc_datetime.date()).replace('-', '_')) file_ in a: if file_ != none: df = pd.read_csv(file_, index_col=none, header=0, delimiter=';') list_.append(df) frame = pd.concat(list_)
Comments
Post a Comment