JREPL.BAT v8.6 - regex text processor with support for text highlighting and alternate character sets

[foxidrive]

wondered if I might be able to use jRepl or FindRepl to take some bulk out of them.

1. One routine parses the output from "ipconfig /all" to get the "DNS Servers". I guess there can be 1 or more, in this case there are three:

Code: Select all

   DNS Servers . . . . . . . . . . . : 192.168.0.1
                                       184.16.4.22
                                       192.168.88.134

2. I have a wordy routine that parses "Local group Membership" from the "NET USER [User Name]" command. The asterik * causes it to be more complex than I'd like. In the example below, I'd like to parse out all of the group names (those beginning with "*test_")

1. There can be more than one set of DNS servers in ipconfig /all
such as belonging to other network items. Are you aware of that?

2. I'm guessing that you edited those and *test_ doesn't appear in the actual data. The characteristics of the text is a common way to filter those lines in batch code and the task isn't clear.

EDIT: There are always other ways to do part 2 but simple code is easier to read and debug into the future.

[dbenham]

1) Parse out DNS Servers from IPCONFIG /ALL

Ignoring Foxidrive's concern about multiple sets, the following shows one way to put all the DNS Servers into an "array", and then demonstrates how to print the first, last, or all values.

Code: Select all

@echo off
setlocal enableDelayedExpansion

for /f "tokens=1,2 delims=: " %%A in (
  'ipconfig /all ^| jrepl "^   DNS Servers (?:\. )+:(([\s\r\n]+(\d+\.){3}\d+)+)" $1 /m /jmatch ^| findstr /n "^"'
) do (
  set "DNSServer.Cnt=%%A"
  set "DNSServer.%%A=%%B"
)

echo(
echo Listing First DNS Server:
echo(!DNSServer.1!

echo(
echo Listing Last DNS Server:
echo(!DNSServer.%DNSServer.Cnt%!

echo(
echo Listing All DNS Servers:
for /l %%N in (1 1 %DNSServer.Cnt%) do echo(!DNSServer.%%N!

Another option is to use the /T option in a similar way as below


2) Parse out all "Local group Membership" groups from the NET USER [User Name] command

Here I use the /T option along with a minimal bit of user supplied JScript code to directly parse out and print the values, one per line. The /JBEG option defines a variable used to indicate whether we are in a section of interest. The first search/replace pair matches the beginning of a "Local Group Memberships" section, as well as the first group, sets the indicator, and prints the first group. The second search/replace pair detects the beginning of any other section (end of previous), and clears the indicator. The last search/replace pair checks the indicator, and only prints the group match if the indicator is TRUE.

Code: Select all

net user user0 | jrepl "^Local Group Memberships|^\w| *\*(\w+)" "go=true;false|go=false|go?$4:false" /jmatch /jbeg "go=false" /t "|"

Dave Benham

[mirrormirror]

1. There can be more than one set of DNS servers in ipconfig /all
such as belonging to other network items. Are you aware of that?

You are right of course - and I did not take this into consideration. For now I think there is only one adapter on each box where this will run. I'll worry about multiple-adapter configurations when/if I need to
--- EDIT---
My prior code did account for multiple adapters. I used the following code to pick an adapter based on IP address, then parse the DNS servers of that particular adapter:

Code: Select all

FOR /F "usebackq tokens=3,4" %a IN (`route print 0.0.0.0 ^|findstr /C:"0.0.0.0"`) DO @ECHO b: %b

Code: Select all

   FOR /F "usebackq tokens=3,4" %%a IN (`route print 0.0.0.0 ^|findstr /C:"0.0.0.0"`) DO (
      SET srcIP=%%b
   )

I'm guessing that you edited those and *test_ doesn't appear in the actual data

It does - these are the names of the groups needing to be parsed:

Code: Select all

Local Group Memberships      *test_et_ro           *test_et_rw
                             *test_t_Links         *test_troot_files
                             *test_sConfigLoc      *test_rs_Temp
                             *test_rs_WAN          *Users

I'm going to try out dbenham's solutions shortly and will report back - Thanky you for the assistance!

[mirrormirror]

Dave, thank you for the examples, they help out a lot. They both work great. I modified the 1st example to pull DNS servers associated with a specific IP:

Code: Select all

FOR /F "usebackq tokens=3,4" %%a IN (`route print 0.0.0.0 ^|findstr /C:"0.0.0.0"`) DO (
   SET srcIP=%%b
)

for /f "tokens=1,2 delims=: " %%A in (
  'type t:\t.txt ^| jrepl "%srcIP%[\s\S]*?^[ \t]*?DNS Servers (?:\. )+:(([\s\r\n]+(\d+\.){3}\d+)+)" $1 /m /jmatch ^| findstr /n "^"'
) do (
  set "DNSServer.Cnt=%%A"
  set "DNSServer.%%A=%%B"
)

The search line changed:

Code: Select all

'ipconfig /all ^| jrepl "%srcIP%[\s\S]*?^[ \t]*?DNS Servers (?:\. )+:(([\s\r\n]+(\d+\.){3}\d+)+)" $1 /m /jmatch ^| findstr /n "^"'
'ipconfig /all ^| jrepl "^   DNS Servers (?:\. )+:(([\s\r\n]+(\d+\.){3}\d+)+)" $1 /m /jmatch ^| findstr /n "^"'

ASSUMPTION #1: This will find the IP address and EVERYTHING up to the line beginning with "DNS Server"

Code: Select all

%srcIP%[\s\S]*?

And I also changed

Code: Select all

^   DNS Servers
to
^[ \t]*?DNS Servers

Because I do not trust IPCONFIG /ALL output to be exactly the same across different OS versions.
ASSUMPTION #2: the code above will find the start of a line then any number of [TABS] or [SPACES] then the string "DNS Servers".
If either of my assumptions are incorrect please let me know.
As it is, it seems to work, but I need to test it out on a few different configurations.

Also, I spotted a very minor typographical issue in your jRepl help:

Code: Select all

::      /N MinWidth            - prefix output with liNe numbers

- notice the capital N in "liNe" - although perhaps it is intentional?

Your 2nd example works but is a bit over my head - I'll work on dissecting it at a later time - but until then, I'll use it in ignorance

Code: Select all

net user user0 | jrepl "^Local Group Memberships|^\w| *\*(\w+)" "go=true;false|go=false|go?$4:false" /jmatch /jbeg "go=false" /t "|"

[dbenham]

Also, I spotted a very minor typographical issue in your jRepl help:

Code: Select all

::      /N MinWidth            - prefix output with liNe numbers

- notice the capital N in "liNe" - although perhaps it is intentional?

It was indeed intentional. I was trying to highlight how I chose the option letters. But I think I should have capitalized "Number" instead

I'll probably change it next time I make an update.


Dave Benham

[mirrormirror]

Ok, I'm back with another question
I'm trying to find an easy/standardized way to generate a few .sql files. Right now there is only one [%var%] that needs to be expanded. In the "printhere" and "heredoc" threads, I've been trying those solutions but none are perfect for what I need. I'm wondering if jrepl can do this?
I would like to FIND everything between the :SQLFile_Begin - :SQLFile_END labels and pipe it out to a new file:

Code: Select all

@ECHO OFF

@ECHO BEGIN ---------------------------------------------
:SQLFile_Begin
      DELETE FROM Users;
      DROP TABLE IF EXISTS tmpDataInsert;
      
      CREATE TEMP TABLE IF NOT EXISTS [tmpDataInsert] ( [uName] CHAR );
      .separator "|"
      .import %DataFile% tmpDataInsert
      
      INSERT INTO Users(UserNm) SELECT * FROM tmpDataInsert;
      DROP TABLE IF EXISTS tmpDataInsert;

:SQLFile_END


@ECHO END ---------------------------------------------

- And the %DataFile% variable would need to be expanded.
I'm having trouble figuring out how to use jrepl for simply "searching" without doing any "replacing".
One more note, I have a few other files I'd like to generate this way - and they do include special characters (> & | ! , etc.).

Thanks for any assistance!

[foxidrive]

I would like to FIND everything between the :SQLFile_Begin - :SQLFile_END labels and pipe it out to a new file:

Code: Select all

@ECHO OFF

@ECHO BEGIN ---------------------------------------------
:SQLFile_Begin
      DELETE FROM Users;
      DROP TABLE IF EXISTS tmpDataInsert;
      
      CREATE TEMP TABLE IF NOT EXISTS [tmpDataInsert] ( [uName] CHAR );
      .separator "|"
      .import %DataFile% tmpDataInsert
      
      INSERT INTO Users(UserNm) SELECT * FROM tmpDataInsert;
      DROP TABLE IF EXISTS tmpDataInsert;

:SQLFile_END


@ECHO END ---------------------------------------------

- And the %DataFile% variable would need to be expanded.

Aacini's findrepl.bat can handle it this way.

Code: Select all

call findrepl :SQLFile_Begin /e::SQLFile_END <file.bat|more|findrepl .DataFile. %DataFile%

I've used a very simple way of determining the variable name, but it will only work for unique text in those lines. Findrepl.bat may already have better way to handle the percent signs.

If the variable names are dynamic then some extra fiddling will be needed.

[dbenham]

I'm having trouble figuring out how to use jrepl for simply "searching" without doing any "replacing".

It is very easy to instruct JREPL to return only what it matches, one match per line. All you need is the /JMATCH option.

Code: Select all

call jrepl "search" $0 /jmatch /f input.txt

But in your case you actually want to do a replacement. You are searching for the begin and end tags, and only keeping what lies between them. If you were to keep the entire match, then the tabs would be included in the output.

You must use the /M option to search across multiple lines.

I pipe the result to a second JREPL to perform the variable replacement. The code below attempts to replace any variable with its value. It leaves any %var% alone if var is not defined.

I'm assuming your code is embedded within a batch script, so I use "%~f0" as the source file.

Code: Select all

jrepl ":SQLFile_Begin\s*?\n([\s\S]*?)\r?\n:SQLFile_END" $1 /m /jmatch /f "%~f0" | jrepl "%%(.*?)%%" "env($1)?env($1):$0" /j /o "output.sql"

I see you have already discovered my PrintHere.bat, but uncovered a bug. Now that I have fixed the bug, it should be very easy to solve your problem with PrintHere.bat. Note that you would want to change %DataFile% into !DataFile! within your source, and use the PrintHere /E option to cause the variable to be expanded in the result.


Dave Benham

[mirrormirror]

jrepl ":SQLFile_Begin\s*?\n([\s\S]*?)\r?\n:SQLFile_END" $1 /m /jmatch /f "%~f0" | jrepl "%%(.*?)%%" "env($1)?env($1):$0" /j /o "output.sql"

This works perfect - thank you.
Now for an advice question: I have some other scenarios where I need to generate output files from within batch files. Some of them have special characters (& > < |, etc.). Can you see any advantages/disadvantages or issues using this jrepl solution for these tasks as compared to using the printhere.bat solution? - other than having to use the /E and !var! to expand variables?
I would like to settle on a solution that works for most or all of these cases without having to tweak it each time.

call findrepl :SQLFile_Begin /e::SQLFile_END <file.bat|more|findrepl .DataFile. %DataFile%

Thanks foxdrive. This seems to work also! If you have any comments on my question to David (above) I'm all ears.

[foxidrive]

I have some other scenarios where I need to generate output files from within batch files. Some of them have special characters (& > < |, etc.).

I'd guess it depends on how those characters are being sent to Jrepl.
If it's from within a file then it's a piece of cake.

[mirrormirror]

I'd guess it depends on how those characters are being sent to Jrepl.

Right now, I am only extracting sections of text from within files. And I am just looking for a consistent, reliable method that will work for all characters. I'd like to avoid tweaking the method and/or text format (i.e. escaping characters, etc.) in the future - I just want a method that will work for my foreseeable needs.

[dbenham]

I don't see any advantage of one solution over another. There shouldn't be any character restrictions for any of JREPL, FINDREPL, or PrintHere.

Use whatever you are most comfortable with.

But why do you need us to tell you that. All you need to do is prepare a file with traditionally "problem" characters, and see how the various solutions handle it.


Dave Benham

[mirrormirror]

But why do you need us to tell you that. All you need to do is prepare a file with traditionally "problem" characters, and see how the various solutions handle it.

I suppose I am just looking to experienced users to advise me of any known limitations/issues I may encounter.
DOS batch character handling has eaten a bunch of my time over the years. I've run tests and such but eventually I will run into a situation that fails
I wish there were a set concise of "RULES" which detailed batch character handling. Things like:
-What characters can be safely passed as parameters to other batch labels
-What characters can be safely passed as parameters to other batch files
- How character handling differs within variables and without variables - ex:

Code: Select all

(ECHO mytext>RandomTextFile)>newTextFile

vs

Code: Select all

SET "myvar=mytext>RandomTextFile"
ECHO %myvar%>newTextFile

- and these are just a few.
My day job is not as a coder and I do these things for personal / friends / family. It can be months between my small scripting sessions because of time constraints and each time I need to dig up some references on how to do stuff because I don't use it every day.

This forum is a tremendous source and the people have been very generous with their time and assistance, but I don't always want to bother others in order to find a solution - and searching/reading through post after post to find an answer is very time consuming.

An example is my current task of generating a few SQL scripts from with batch files. I want the code to be as simple as possible, readable, and maintainable. I really don't like having to escape every parentheses "^( ... ^)" if I can avoid it, because when I change the SQL script, I will probably miss one of these instances and waste time trying to figure out why the whole process failed. I did my searches and read through several solutions here on dostips as well as SO. I found your printhere.bat, as well as a couple of "heredoc" solutions - none of them worked out of the box for me - but they are working now. you guys provided me with some solutiuons using jrepl & findrepl and so I have several viable solutions and just looking for advice on the best one for my tasks.

Please don't take my questions for advice as an offloading of due dilligence. I've learned over the years that there is great knowledge & wisdom in groups and so I ask for advice much more now than when I was younger and "wiser" - in my own eyes.
As it stands right now, I'm preferring the jrepl solution since I don't have to do anything special with %variable% names or exclamation point escaping "^!" - I am very appreciative for you help in providing both the "jrepl.bat" as well as the syntax for my needs - thank you.

[foxidrive]

I wish there were a set concise of "RULES" which detailed batch character handling. Things like:
-What characters can be safely passed as parameters to other batch labels
-What characters can be safely passed as parameters to other batch files
- How character handling differs within variables and without variables - ex:

I see your point very clearly and all the regulars will probably agree that there is nothing concise about batch files rules except the rule that says "Everything has it's own set of rules"

Jrepl will handle all standard characters from a file with no need for any quoting. Essentially it doesn't use batch scripting for the jrepl engine at all, which makes it so robust.

Findrepl is in the same category and Dave's other tools and they all use Jscript which is only launched by a batch file.

[mirrormirror]

Ok, I'm back with another problem This one is not crucual as I have a work-a-round but it is far from ideal. This is more of a jscript regex question.

I want to parse the output from this:

Code: Select all

C:\>icacls "c:\Program Files\Common Files"
c:\Program Files\Common Files NT SERVICE\TrustedInstaller:(F)
                              NT SERVICE\TrustedInstaller:(CI)(IO)(F)
                              NT AUTHORITY\SYSTEM:(M)
                              NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
                              BUILTIN\Administrators:(M)
                              BUILTIN\Administrators:(OI)(CI)(IO)(F)
                              BUILTIN\Users:(RX)
                              BUILTIN\Users:(OI)(CI)(IO)(GR,GE)
                              CREATOR OWNER:(OI)(CI)(IO)(F)

I just want all of the users/privs - basically everything except the <folder path> and <whitespace>. The only problem I have is the first line of output:
c:\Program Files\Common Files NT SERVICE\TrustedInstaller:(F)
I only want the GREEN - not the RED. The problem is the "\" in the file paths - they mess up a [literal] search string in regex - and I suppose periods (.) would do the same. I guess the \Q...\E syntax allowed by other regex engines would solve this but as it stands I don't know how to parse that out. This needs to work for an unknown number of <folder paths> which may include [spaces], [!], etc. I'd like to do something like this but it doesn't work:

Code: Select all

FOR %%c IN  ("c:\Program Files (x86)"; "c:\Program Files\Common Files") DO (
   FOR /F "usebackq tokens=* delims=" %%d IN (`icacls "%%~c" ^|jrepl "(%%~c)?(.*)" $2 /jmatch`) DO @ECHO %%d
)

My work-a-round is to get the length of the <folder path> (I think the :strlen code is from jeb) then simply skip those number of characters:

Code: Select all

@ECHO OFF
FOR %%c IN  ("c:\Program Files (x86)"; "c:\Program Files\Common Files") DO (
   CALL :testLabel %%~c
   FOR /F "usebackq tokens=* delims=" %%d IN (`icacls "%%~c" ^|jrepl "%%~c(.*)" $1 /jmatch`) DO @ECHO %%d
)
GOTO :theEnd

:testLabel
   SET mytmpFldr=%*
   CALL :strLen myLen mytmpFldr
   SET /A myLen+=1
   CALL ECHO CurrentFolder: __________ %*
   FOR /F "usebackq tokens=* delims=" %%d IN (`icacls "%*" ^|jrepl ".{%myLen%}(.*)$" $1 /jmatch`) DO (
      @ECHO %%d
   )
   @ECHO(
GOTO:EOF

:strlen <resultVar> <stringVar>
(   
    setlocal EnableDelayedExpansion
    set "s=!%~2!#"
    set "len=0"
    for %%P in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do (
        if "!s:~%%P,1!" NEQ "" ( 
            set /a "len+=%%P"
            set "s=!s:~%%P!"
        )
    )
)
( 
    endlocal
    set "%~1=%len%"
    exit /b
)

:theEnd

The reason I make the CALL :testLabel %%~c is because I don't want to use DELAYED expansion in my FOR loop - to avoid problems with Exclamation marks (!) in folder names.

Any ideas?