DosTips.com

Hello All,

I found this form by looking for a better way to parse logs and someone advised to use JREPL7.0. The issue here is I am new and a bit confused to advance regex. What I need is a syntax to scan large log files and move log lines that don't have date/time at the beginning of the file to the logline above.

In my log files, log lines always start with entries like "2017-08-16T09:25:12.521-0400" however, when we hit exceptions, xml data, sql query results, it's printed under the log lines with no time stamp.

I am trying to create a JREPL command to take the data with no time stamps and add them on the lines with the above timestamp. The Goal here is to end up with a log that has only lines that start with the date/time stamps. This way, I can merge other logs and sort them by time.




End Result:
2017-04-24T04:26:58.728-0400 - VALUE {tomcat-device-http-1} none|none [{{0ecfe8a7}{Uri, - WARN Could not marshal entity java.lang.ArrayIndexOutOfBoundsException: -1 at com.sun..java:487) at com.sun:323) at com.sun.java:251)
2017-04-24T04:26:59.425-0400 VALUE {tomcat-device-http-1} none|none
2017-08-16T09:25:12.568-0400 - bla

Using Windows 10 machines to do this

To start with, you will need version 7.1 instead of the bugged 7.0. (Or you could use v6.8 since you don't need the v7 features. But I recommend 7.1)

Since you need to remove new line characters, you will need to use the /M option, which limits the solution to files no bigger than ~1GB. There is a way to process larger files line by line, but that would require a bit of user supplied JScript code (easily done, but I'm assuming you don't need this)

The following should work
I match the entire time stamp, all the way through the time zone. You could probably get away with a significantly shorter test which would shorten the command.

If you put the command within another batch script, then you must use CALL JREPL.

You can make things look a little prettier (and easier to follow) if you define a variable containing the regex to match an unwanted new line.


Dave Benham

Thanks, it seems to be working. can you explain the syntax here? trying to better understand how its doing what its doing

Just another question for you. Now that i have xx amounts of logs done with the above do you have any javascript or other technics to combine the logs, sort and split them again?

1. combine all logs to one file
2. sort file by timestamp
3. delete dup lines
4. split log by size


1. Any better ways to grep data from multiple files to one?




2. for #2, sort does not like the the size of the file from #1



Edit: Found JSORT but its currently using 9GB memory for 3.5GB file