Page 1 of 1
Looking for some help greping a large logfile and split the logs in separate logs per hour
Posted: 22 Sep 2017 10:05
by rcmpayne
Looking to use JREPL to grep a very large file (5-15GB or more) file and split the file to per hour. the example below would be the start of every logline.
2017-04-24T01:26:58.728-0400 -
2017-04-24T03:26:58.728-0400 -
2017-04-24T03:26:58.728-0400 -
2017-04-24T04:26:58.728-0400 -
2017-04-24T04:26:58.728-0400 -
2017-04-24T05:26:58.728-0400 -
2017-04-24T05:26:58.728-0400 -
Re: Looking for some help greping a large logfile and split the logs in separate logs per hour
Posted: 22 Sep 2017 10:26
by Squashman
If every line starts with a date and time stamp, you certainly wouldn't need to use JREPL to do that.
Re: Looking for some help greping a large logfile and split the logs in separate logs per hour
Posted: 25 Sep 2017 09:56
by rcmpayne
Trying something like this works but not very sufficient. any tips would be great
set InputFile=ALL_UOS.txt
set OutputFile=ALL_UOS.txt
Cant get this to work
Code: Select all
rem set "CoreRegex=\r?\n(?!\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d\d\d-\d{4})"
rem cat ALL_UOS_ACCS.txt | grep \d{4}-\d\d-\d\dT05:\d\d:\d\d\.\d\d\d-\d{4}
rem cat ALL_UOS_ACCS.txt | grep -E "\d{4}-\d\d-\d\dT05:\d\d:\d\d\.\d\d\d-\d{4}"
These work if i run one for every time hour (T00, T01-T23)
Code: Select all
cat %InputFile% | grep T00: > T00_%OutputFile%
cat %InputFile% | grep T01: > T01_%OutputFile%
cat %InputFile% | grep T02: > T02_%OutputFile%
cat %InputFile% | grep T03: > T03_%OutputFile%
cat %InputFile% | grep T04: > T04_%OutputFile%
cat %InputFile% | grep T05: > T05_%OutputFile%
cat %InputFile% | grep T06: > T06_%OutputFile%
cat %InputFile% | grep T07: > T07_%OutputFile%
cat %InputFile% | grep T08: > T08_%OutputFile%
cat %InputFile% | grep T09: > T09_%OutputFile%
cat %InputFile% | grep T10: > T10_%OutputFile%
cat %InputFile% | grep T11: > T11_%OutputFile%
...