JREPL command to move loglines that dont start with a date/timestamp to the line above that has a date/timestamp

[rcmpayne]

Hello All,

I found this form by looking for a better way to parse logs and someone advised to use JREPL7.0. The issue here is I am new and a bit confused to advance regex. What I need is a syntax to scan large log files and move log lines that don't have date/time at the beginning of the file to the logline above.

In my log files, log lines always start with entries like "2017-08-16T09:25:12.521-0400" however, when we hit exceptions, xml data, sql query results, it's printed under the log lines with no time stamp.

I am trying to create a JREPL command to take the data with no time stamps and add them on the lines with the above timestamp. The Goal here is to end up with a log that has only lines that start with the date/time stamps. This way, I can merge other logs and sort them by time.

Code: Select all

2017-04-24T04:26:58.728-0400 - VALUE {tomcat-device-http-1} none|none 
[{{0ecfe8a7}{Uri, - WARN Could not marshal entity
java.lang.ArrayIndexOutOfBoundsException: -1
    at com.sun..java:487)
    at com.sun:323)
    at com.sun.java:251)
2017-04-24T04:26:59.425-0400 VALUE {tomcat-device-http-1} none|none 
2017-08-16T09:25:12.568-0400 - bla

End Result:
2017-04-24T04:26:58.728-0400 - VALUE {tomcat-device-http-1} none|none [{{0ecfe8a7}{Uri, - WARN Could not marshal entity java.lang.ArrayIndexOutOfBoundsException: -1 at com.sun..java:487) at com.sun:323) at com.sun.java:251)
2017-04-24T04:26:59.425-0400 VALUE {tomcat-device-http-1} none|none
2017-08-16T09:25:12.568-0400 - bla

Using Windows 10 machines to do this

[dbenham]

To start with, you will need version 7.1 instead of the bugged 7.0. (Or you could use v6.8 since you don't need the v7 features. But I recommend 7.1)

Since you need to remove new line characters, you will need to use the /M option, which limits the solution to files no bigger than ~1GB. There is a way to process larger files line by line, but that would require a bit of user supplied JScript code (easily done, but I'm assuming you don't need this)

The following should work

Code: Select all

jrepl.bat "\r?\n(?!\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d\d\d-\d{4})(?:\t| |\r?\n(?!\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d\d\d-\d{4}))*" " " /m /f "input.txt" /o -

I match the entire time stamp, all the way through the time zone. You could probably get away with a significantly shorter test which would shorten the command.

If you put the command within another batch script, then you must use CALL JREPL.

You can make things look a little prettier (and easier to follow) if you define a variable containing the regex to match an unwanted new line.

Code: Select all

@echo off
setlocal
set "BadNewLine=\r?\n(?!\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d\d\d-\d{4})"
call jrepl.bat "%BadNewLine%(?:\t| |%BadNewLine%)*" " " /m /f "input.txt" /o -

Dave Benham

[rcmpayne]

Thanks, it seems to be working. can you explain the syntax here? trying to better understand how its doing what its doing

[rcmpayne]

Just another question for you. Now that i have xx amounts of logs done with the above do you have any javascript or other technics to combine the logs, sort and split them again?

1. combine all logs to one file
2. sort file by timestamp
3. delete dup lines
4. split log by size


1. Any better ways to grep data from multiple files to one?

Code: Select all

echo Copy all other logs with timestamp
FOR /f %%b IN ('dir %Folder%\*CORE*.* /b /s') do (grep ^2017- %%b >> %Results_ALL_TMP%)

2. for #2, sort does not like the the size of the file from #1

Code: Select all

echo Sort the massive log
sort -k1 %Results_ALL_TMP% > %Results_ALL%

Edit: Found JSORT but its currently using 9GB memory for 3.5GB file

Code: Select all

SortScript.bat newfile.txt /O newfile_sort.txt