FindRepl.bat:New regex utility to search and replace strings

[Newbee]

I have this batch file:

Code: Select all

@echo off
echo Show both the first 10 lines and the last 10 lines (with line numbers):
echo(
< input1.txt FindRepl /V /O:10:-10 /N

"input1.txt" file

Code: Select all

Line 1
Lines 1 extra
Line 2
Lines 2 extra
Line 3
Lines 3 extra
Line 4
Lines 4 extra
Line 5
Lines 5 extra
Line 6
Lines 6 extra
Line 7
Lines 7 extra
Line 8
Lines 8 extra
Line 9
Lines 9 extra
Line 10
Lines 10 extra
Line 11
Line 11 extra
Line 12
Line 12 extra
Line 13
Line 13 extra
Line 14
Line 14 extra
Line 15
Line 15 extra

It just display 9 lines from head and 9 lines from tail only. Here is the output:

Code: Select all

d:\4\FindRepl>ex7
Show both the first 10 lines and the last 10 lines (with line numbers):

1:Line 1
2:Lines 1 extra
3:Line 2
4:Lines 2 extra
5:Line 3
6:Lines 3 extra
7:Line 4
8:Lines 4 extra
9:Line 5
22:Line 11 extra
23:Line 12
24:Line 12 extra
25:Line 13
26:Line 13 extra
27:Line 14
28:Line 14 extra
29:Line 15
30:Line 15 extra

d:\4\FindRepl>

[Aacini]

Yes. This is the "input1.txt" file with line numbers:

Code: Select all

 1:Line 1
 2:Lines 1 extra
 3:Line 2
 4:Lines 2 extra
 5:Line 3
 6:Lines 3 extra
 7:Line 4
 8:Lines 4 extra
 9:Line 5
10:Lines 5 extra
11:Line 6
12:Lines 6 extra
13:Line 7
14:Lines 7 extra
15:Line 8
16:Lines 8 extra
17:Line 9
18:Lines 9 extra
19:Line 10
20:Lines 10 extra
21:Line 11
22:Line 11 extra
23:Line 12
24:Line 12 extra
25:Line 13
26:Line 13 extra
27:Line 14
28:Line 14 extra
29:Line 15
30:Line 15 extra

The description of /O:s:e switch specify: "Show block of lines, from line S to line E... if E is negative, it specifies counting lines from the end of file... the last line of the file is -1". In this way, the /O:10:-10 switch means show the block of numbered lines from the line number 10 up to the line number -10, that in this case is the same as line 21. That is to say, the /O:10:-10 switch alone would show the block of indented lines below:

Code: Select all

   1:Line 1
   2:Lines 1 extra
   3:Line 2
   4:Lines 2 extra
   5:Line 3
   6:Lines 3 extra
   7:Line 4
   8:Lines 4 extra
   9:Line 5
       10:Lines 5 extra
       11:Line 6
       12:Lines 6 extra
       13:Line 7
       14:Lines 7 extra
       15:Line 8
       16:Lines 8 extra
       17:Line 9
       18:Lines 9 extra
       19:Line 10
       20:Lines 10 extra
-10    21:Line 11
-9 22:Line 11 extra
-8 23:Line 12
-7 24:Line 12 extra
-6 25:Line 13
-5 26:Line 13 extra
-4 27:Line 14
-3 28:Line 14 extra
-2 29:Line 15
-1 30:Line 15 extra

The /V switch specify: "Show non-matching lines", that is, show the lines of the file excepting the block of lines from line 10 up to line 21. The result is two block of lines: from line 1 to line 9, and from line 22 to the last line.

In the first post of this thread there is this example:

2. Finding blocks of lines (/E and /O switches)

- Show both the first 15 lines and the last 20 lines (with line numbers):

Code: Select all

< theFile.txt FindRepl /V /O:16:-21 /N

Antonio

[Newbee]

Thanks for the clarification Antonio.
It's my fault when not apply the example correctly. I've just read this line
" Show both the first 15 lines and the last 20 lines (with line numbers):" and
just go ahead to plug in the same values in the example code.
My apology for that.

[Newbee]

I have this batch file:

Code: Select all

@echo off
call FindRepl "D:\1" /S:="%PATH%" > NUL
echo errorlevel = %errorlevel%
if %errorlevel% gtr 0 echo The given path exists in system PATH

My %errorlevel% is always 0, even though that "D:\1" is in my path.

Here is my path:

Code: Select all

"D:\1;D:\2;D:\gcc\bin;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;"

[Squashman]

Yes. Zero means it found it. Same errorlevel you would get if you were using the FIND and FINDSTR commands. If it finds the search string, the errorlevel is set to 0. If it doesn't find the search string the errorlevel is set to 1.

[Aacini]

3. Basic usage of regular expressions

The Search string is not plain text, but a regular expression ("regexp"). In its most basic form, a regexp just includes the literal string to search: "Search this".

Several features may be included in a regexp, most of them selected via a backslash character: \. You may use a backslash followed by another character to specify binary bytes; for example, to specify a <TAB> character you may use \t or \cI (Ctrl-I) or \x09 (Ascii hexa-code 9). For a <LF> (newline) character you may use \n or \cJ or \x0A, and use \r or \cM or \x0D for <CR> (return).

Certain combinations of \ and the next character allows to specify sets of common characters. For example, use \d for any digit (equivalent to [0-9]), \D for nondigit characters (equivalent to [^0-9]), \w for alphanumeric characters and underscore ([A-Za-z0-9_]), \W for the rest of special characters, \s for separators (space, tab, return, new-line) and \S for non-separators. A dot alone match any character except a newline; to search for a dot, precede it with backslash.

You may anchor the beginning of line with ^ or the end of line with $, or use \b to anchor the beginning or end of a word. An asterisk repeat the previous matched character zero or more times, so "\d*" match an optional number. A plus sign repeat previous match one or more times ("\d+" match a required number), and a question mark repeat zero or one time ("\d?" match an optional digit).

For a complete description of this topic, see: http://msdn.microsoft.com/en-us/library/1400241x(v=vs.84).aspx Please note that you always must escape with backslash the following special characters in order to search for themselves: \*+?^$.[]{}()|

Or, in shorter form from the FindRepl /? help screen:

Code: Select all

All search texts must be given in VBScript regular expression format (regexp).
Matching characters: .=any char, \d=[0-9], \D=non-digit, \w=[A-Za-z0-9_],
\W=non-alphanumeric, \s=[ \t\r\n], \S=non-space. Anchors: ^=begin of line,
$=end of line, \b=begin/end of word. Quantifiers (repeats previous match):
*=zero or more times, +=one or more times, ?=zero or one time. Use parentheses
to delimit these constructs and create "subexpressions". To search for these
special characters, place a back-slash before each one of them: \*+?^$.[]{}()|

Also:

Code: Select all

/S:sSource Text to be processed instead of Stdin file.

If the first character of any text is an equal-sign, it specifies the name of
a Batch variable that contain the text to be processed.

Use:

Code: Select all

call FindRepl "D:\\1" /S:"%PATH%" > NUL

or:

Code: Select all

call FindRepl "D:\\1" /S:=PATH > NUL

Please, read the documentation...

Antonio

[Aacini]

Yes. Zero means it found it. Same errorlevel you would get if you were using the FIND and FINDSTR commands. If it finds the search string, the errorlevel is set to 0. If it doesn't find the search string the errorlevel is set to 1.

Code: Select all

The total number of matchings/replacements is returned in ERRORLEVEL.

[Newbee]

I do this:

Code: Select all

FindRepl "D:\\1" /S:=D:\1;D:\2; >NUL

or this:

Code: Select all

FindRepl "D:\\1" /S:="D:\1;D:\2;" >NUL

The %errorlevel% are always 0, is it right?

[Aacini]

Code: Select all

C:\Users\Antonio\Tests> FindRepl "D:\\1" /S:"D:\1;D:\2;" >NUL

C:\Users\Antonio\Tests> echo %errorlevel%
1

C:\Users\Antonio\Tests> set "VAR=D:\1;D:\2;"

C:\Users\Antonio\Tests> FindRepl "D:\\1" /S:=VAR >NUL

C:\Users\Antonio\Tests> echo %errorlevel%
1

The format of /S switch is:

Code: Select all

/S:"string"

Or

/S:=Variable

If you use FindRepl "D:\\1" /S:=D:\1;D:\2; >NUL you are searching the string D:\1 in a variable named D:\1;D:\2;.

If you use FindRepl "D:\\1" /S:="D:\1;D:\2;" >NUL you are searching the string D:\1 in a variable named "D:\1;D:\2;".


To search in a string, enclose the string in quotes and do not include the equal-sign: /S:"literal string" or /S:"%Variable%".

To search in a variable, include the equal sign and the variable name; do not include quotes nor percent-signs: /S:=Variable.

[thefeduke]

I do this:

Code: Select all

FindRepl "D:\\1" /S:=D:\1;D:\2; >NUL

. . .
The %errorlevel% are always 0, is it right?

Walk before running. Perhaps it would be easier to correlate the errorlevel with the results if you did not suppress them. Get rid of >NUL and you can see for yourself if there was output.

John A.

[Newbee]

Thanks for the clarification Antonio.
I do this:

Code: Select all

call FindRepl "D:\\1" /S:"%PATH%" > NUL
echo errorlevel = %errorlevel%
if %errorlevel% gtr 0 echo The given path exists in system PATH

or this:

Code: Select all

call FindRepl "D:\\1" /S:=PATH > NUL
echo errorlevel = %errorlevel%
if %errorlevel% gtr 0 echo The given path exists in system PATH

They now both give %errorlevel% = 1.

This is really cool.

[MrWinelover]

I need to manipulate a text file so that a part of each line is replaced (or deleted).
The first part that must replaced (or deleted) is situated between first and second last occurence of the piping character.
Another part that must replaced (or deleted) is situated between first and last occurence of minus character.

I have already gone through the description of FindRepl.bat but could not immediately find out how to achieve this.
Is it possible to do this with FindRepl.bat?

The text file itselfs is already generated by manipulating another (log) file with the command findstr to keep only the lines we need.
Can the replace command be combined by the findstr command?

Example:
Original strings:
18/07/2018 06:36:42|HARDWARE|Base|send|ERROR|10.15.149.65 No communication Counter set-global-level1-occured
18/07/2018 06:42:07|HARDWARE|Base|send|INFO|10.15.149.65 Communication restored Counter reset-global-level1-fixed
18/07/2018 10:17:03|HARDWARE|Base|send|ERROR|10.15.149.65 No working position Counter set-main board-level2-occured
18/07/2018 10:24:19|HARDWARE|Base|send|INFO|10.15.149.65 Normal state Counter reset-main board level2-occured

Wanted result after manipulation thru Batch file
18/07/2018 06:36:42|ERROR|10.15.149.65 No communication Counter set-occured
18/07/2018 06:42:07|INFO|10.15.149.65 Communication restored Counter reset-fixed
18/07/2018 10:17:03|ERROR|10.15.149.65 No working position Counter set-occured
18/07/2018 10:24:19|INFO|10.15.149.65 Normal state Counter reset-occured

[Squashman]

Why not just use plain old FOR /F commands?

Code: Select all

@echo off

(FOR /F "TOKENS=1,5,6 DELIMS=|" %%G IN (input.txt) DO (
	FOR /F "TOKENS=1,4 DELIMS=-" %%J IN ("%%I") DO echo %%G^|%%H^|%%J-%%K)
)>output.txt

or

Code: Select all

@echo off
setlocal enabledelayedexpansion
(FOR /F "TOKENS=1,5,6 DELIMS=|" %%G IN (input.txt) DO (
	FOR /F "TOKENS=1* DELIMS=-" %%J IN ("%%I") DO (
		set "last=%%K"
		set "last=!last:-= !"
		FOR %%M IN (!last!) DO set "last=%%M"
		echo %%G^|%%H^|%%J-!last!)
	)
)>output.txt

[MrWinelover]

Thank you for the quick answer, but I would have liked some additional information as I have some knowledge but I am not an expert.
Due to insufficient knowledge, I have never used this sytax before.
As far as I can analyze, this only works if the exact positions are known in advance. Correct?
From where do the %% variables come from?
How are they determined?
It is mainly in both examples the piece of code after the second DO that I do not understand
Can you explain the code a.u.b?

[dbenham]

The first code only works if the number of pipes is constant and the number of dashes is constant.

The second code still requires the number of pipes to be constant, but the number of dashes can vary.

A variable like %%G within a batch file is a FOR variable. If used on the command line, then it would be %G instead.

FOR loops can be very confusing for beginners because slight changes in syntax can result in totally different function. Getting the exact behavior you want is often difficult and/or tedious even for experienced users.

A simple FOR iterates strings and or files, depending on whether there are wildcards within the IN() clause.

FOR /F parses one or more tokens from each line that it iterates, with a different %%_ variable for each token. The lines that are iterated can either be the content of a file, the output of a command, or a string literal, depending on the syntax used within the IN() clause.

You should carefully read the help text that you get when you execute FOR /? from the command line and try some experiments on your own.

Both of Squashman's codes could be combined with your FINDSTR and eliminate your intermediate file if you change the outer FOR /F loop as follows:

Code: Select all

(FOR /F "TOKENS=1,5,6 DELIMS=|" %%G IN ('findstr "yourSearchHere" input.log') DO (

However, if the output of FINDSTR is huge, then it is much faster to preserve the two steps and use the temporary intermediate file.

===============================

Aacini's FINDREPL is a sophisticated utility. I'm sure it could easily meet your requirements, but I haven't taken the time to learn how to use FINDREPL.

My JREPL.BAT utility serves a similar purpose, but with totally different syntax.

JREPL can solve your problem with a single, fairly short one liner:

Code: Select all

call jrepl "\|.*(\|.*?\|) -.*-" "$2 -" /t " " /inc "/XXX/" /a /f original.log /o output.txt

The first argument represents a pair of regular expression search terms, delimited by a space. The second argument specifies the corresponding replacement terms, again delimited by a space. The /T " " option is what dictates that the arguments are space delimited lists of find/replace terms.

The capture group numbering is a bit tricky because each search term is converted into its own captured group behind the scenes. The explicit captured group in the first term is actually group 2 because the first term is group 1. The second term becomes group 3.

The /INC "/XXX/" option specifies which lines should be included in the FIND/REPLACE operation. The XXX represents a regular expression that you would have to supply to take the place of the FINDSTR that you are currently using.

The /A option causes only lines that have been altered to be output, thus preventing the lines that don't match the /INC regular expression to be excluded from the output.

The /F original.log option specifies your input, and /O output.txt specifies your output.

If you want to continue using FINDSTR to filter the lines, then you only need the following

Code: Select all

call jrepl "\|.*(\|.*?\|) -.*-" "$2 -" /t " " /f input.txt /o output.txt

If you have additional questions regarding Squashman's code, or my JREPL solution, then I suggest you start a new topic. I don't want to hijack Aacini's FINDREPL topic anymore than I already have.


Dave Benham