JREPL.BAT v8.6 - regex text processor with support for text highlighting and alternate character sets

[naraen87]

@dbenham

I've created the my bat file as follows to edit my property is that ok

Code: Select all

@echo off
if not exist "PropertyFile.txt" goto :EOF
if not exist "%~dp0jrepl.bat" goto :EOF

call "%~dp0jrepl.bat" "^(Environment=.+)$" "Environment=UAT1-v6.19.13.0" /I /F "PropertyFile.txt" /O -

rem JREPL.BAT v7.9 leaves behind JREPL\XBYTES.HEX in folder for temporary files.
rd /Q /S "%TEMP%\JREPL" 2>nul

[dbenham]

Looks good

Currently the parentheses are not doing anything useful, but they are doing no harm either. It would work just as well if you did:

Code: Select all

call "%~dp0jrepl.bat" "^Environment=.+$" "Environment=UAT1-v6.19.13.0" /I /F "PropertyFile.txt" /O -

Or, with parentheses, you can save a bit of typing - $1 represents the value matched in the first (only) set of parentheses:

Code: Select all

call "%~dp0jrepl.bat" "^(Environment=).+$" "$1UAT1-v6.19.13.0" /I /F "PropertyFile.txt" /O -

Dave Benham

[naraen87]

Hi Dave
I'm running my command through a bat file and how could I confirm whether my change is happened or not. Is any errorlevel are you following in JREPL.

Is the following scenario handled in your bat file
I'm running the below command to change a value in the property file

Code: Select all

E:\Jenkins_Software\PSTools\PsExec.exe \\10.47.36.182 call "\\10.47.36.182\C$\Users\boonar\Desktop\testscripts\JREPL7.9\JREPL.bat" "^(Environment=.+)$" "Environment=UAT1-v6.19.21.0" /I /F "\\10.47.36.182\C$\narayana\Enviromnet_Setup\WIServerSetup\UAT\MDrive\BaNCSFS\BancsProduct\Intranet\properties\InputFiles\MCSysProp.properties" /O -

In the meantime I'm doing some change in the file by manually open that file and edit through Notepad.
In that time the above command will return error or success. (I guess it needs to return some error or it needs to do the change with my manual change)

[dbenham]

A full description of error return codes, like everything else, is included in the built in help.

To get a description of the return codes:

Code: Select all

prompt>jrepl /?return

  Possible ERRORLEVEL Return Codes:

      If /? was used, and no other argument
          0 = Only possible return

      If /MATCH, /JMATCH, /JMATCHQ, /K, and /R were not used
          0 = At least one change was made
          1 = No change was made
          2 = Invalid call syntax or incompatible options
          3 = JScript runtime error

      If /MATCH, /JMATCH, /JMATCHQ, /K, or /R was used
          0 = At least one line was written
          1 = No line was written
          2 = Invalid call syntax or incompatible options
          3 = JScript runtime error

To get a description of all available help:

Code: Select all

prompt>jrepl /?help

  Help is available by supplying a single argument beginning with /? or /??:

      /?        - Writes all available help to stdout.
      /??       - Same as /? except uses MORE for pagination.

      /?Topic   - Writes help about the specified topic to stdout.
                  Valid topics are:

                    INTRO   - Basic syntax and default behavior
                    OPTIONS - Brief summary of all options
                    JSCRIPT - JREPL objects available to user JScript
                    RETURN  - All possible return codes
                    VERSION - Display the version of JREPL.BAT
                    HISTORY - A summary of all releases
                    HELP    - Lists all methods of getting help

                  Example: List a summary of all available options

                     jrepl /?options

      /?WebTopic - Opens up a web page within your browser about a topic.
                  Valid web topics are:

                    REGEX   - Microsoft regular expression documentation
                    REPLACE - Microsoft Replace method documentation
                    UPDATE  - DosTips release page for JREPL.BAT
                    CHARSET - List of possible character set names for ADO I/O
                              Some character sets may not be installed
                    XREGEXP - xRegExp.com home page (extended regex docs)

      /?/Option - Writes detailed help about the specified /Option to stdout.

                  Example: Display paged help about the /T option

                     jrepl /??/t

      /?CHARSET/[Query] - List all character set names for use with ADO I/O
                  that are installed on this computer. Optionally restrict
                  the list to names that contain Query. Wildcards * and ? may
                  be used within Query. The default Query is an empty string,
                  meaning list all available character sets. The list is
                  generated via reg.exe.

                  Examples:

                     jrepl /??charset/    - Paged list of all available names
                     jrepl /?charset/utf  - List of names containing "utf"

Dave Benham

[batnoob]

I am learning to use JavaScript, and JREPL is now very interesting to me. I read your code and *most* of your documentation, but I am still confused about how you called in the javascript portion of your code in a batch file. I would appreciate if you could clarify.

Thanks,
BatNoob.

[aGerman]

In post #10 of the following thread I explained it
viewtopic.php?t=7209&p=46984#p46984
But there are various threads about hybrid scripts on DosTips to learn more.

Steffen

[muppe1970]

Is there any way of removing the BOM from the UTF-8 output files? I am reading in an utf-8 file, do some find & replace and write an output file which should also be an utf-8 file but without BOM. I tried to redirect the stdio to a file but it does not create the proper utf-8 file.

Marko

[dbenham]

Unfortunately ADO always writes UTF-8 with BOM. At one point I researched what it would take to control whether the BOM was written or not, and I wasn't able to come up with a clean solution. I may revisit this in the future and try to come up with a solution.

In the mean time, there is a JREPL hack you can use to remove the BOM after the output has been created - simply treat the output file as binary, and remove the BOM with another JREPL:

Code: Select all

call jrepl "^\xEF\xBB\xBF" "" /inc 1 /xseq /m /f "yourOutputFile.txt" /o -

If your machine does not default to a single byte character set, then you need to explicitly force the use of a single byte character set - windows-1252 should work.

Code: Select all

call jrepl "^\xEF\xBB\xBF" "" /inc 1 /xseq /m /f "yourOutputFile.txt|windows-1252" /o -

Dave Benham

[penpen]

The only way to the bom from stream within JScript (as "SaveToFile" copies the whole stream content ignoring the actual stream "Position", untested):

Code: Select all

@if (true==false) @end /*
@echo off
setlocal enableExtensions disableDelayedExpansion
cscript //E:JScript //Nologo "%~f0" /bom:1 
cscript //E:JScript //Nologo "%~f0" /bom:0 
goto :eof
*/

var bom = WScript.Arguments.Named.Exists("bom") ? (WScript.Arguments.Named.Item("bom") == 1) : false;
var filename = (bom) ? ".\\bom.txt" : ".\\noBom.txt";
var stream = WScript.CreateObject("ADODB.Stream");

stream.Open();
stream.Charset = "UTF-8";
stream.WriteText("\u00e4\u00f6\u00fc");

if (bom) {
	stream.SaveToFile(filename, 2);                         // adSaveCreateOverWrite = 2
	stream.Flush();
} else {
	var noBomStream = WScript.CreateObject("ADODB.Stream");
	noBomStream.Type = 1;                                   // adTypeBinary = 1
	noBomStream.Mode = 3;                                   // adModeReadWrite = 3
	noBomStream.Open();

	stream.Position = 3;                                    // skip bom
	stream.CopyTo(noBomStream);

	noBomStream.SaveToFile(filename, 2);                    // adSaveCreateOverWrite = 2
	noBomStream.Flush();
	noBomStream.Close();
}

stream.Close();

penpen

[dbenham]

Yes, I had seen that technique, but I haven't seen a good way to automatically detect when I'm dealing with an encoding that involves a BOM, and what is the size of the BOM. The JREPL /O (output) option lets you choose the encoding (character set). I'm thinking I'd have to hard code recognition of certain character set strings, and I am reluctant to do that.

Dave Benham

[penpen]

Actually i'm unsure, if you could get the bom if you "loadFromFile", but if you create a new stream, then you could do something like that:

Code: Select all

@if (true==false) @end /*
@echo off
setlocal enableExtensions disableDelayedExpansion
cscript //E:JScript //Nologo "%~f0" /bom:1 
cscript //E:JScript //Nologo "%~f0" /bom:0 
goto :eof
*/

var bom = WScript.Arguments.Named.Exists("bom") ? (WScript.Arguments.Named.Item("bom") == 1) : false;
var filename = (bom) ? ".\\bom.txt" : ".\\noBom.txt";
var stream = WScript.CreateObject("ADODB.Stream");
var bomBytes = null;
var bomString = "";
var bomSize = 0;


stream.Open();
stream.Charset = "UTF-8";

if (bom) {
	stream.WriteText("");                                 // force bom creation
	bomSize = stream.Size;                                // get bom size

	bomStream = WScript.CreateObject("ADODB.Stream");     // copy bom to iostream
	bomStream.Type = 1;                                   // adTypeBinary = 1
	bomStream.Mode = 3;                                   // adModeReadWrite = 3
	bomStream.Open();
	stream.Position = 0; stream.CopyTo(bomStream);
	stream.Position = bomSize;

	bomStream.Position = 0;                               // read bom from stream
	bomBytes = bomStream.Read (bomSize);
	bomStream.Close();

	hex = new ActiveXObject("microsoft.xmldom").createElement("bomBytes"); // read hex
	hex.dataType = "bin.hex";
	hex.nodeTypedValue = bomBytes;
	bomString = hex.text;
}

WScript.Echo("bomString = [" + bomString + "]");              // echo result
WScript.Echo("bomSize = " + bomSize);


stream.WriteText("\u00e4\u00f6\u00fc");

if (bom) {
	stream.SaveToFile(filename, 2);                         // adSaveCreateOverWrite = 2
	stream.Flush();
} else {
	var noBomStream = WScript.CreateObject("ADODB.Stream");
	noBomStream.Type = 1;                                   // adTypeBinary = 1
	noBomStream.Mode = 3;                                   // adModeReadWrite = 3
	noBomStream.Open();

	stream.Position = bomSize;                              // skip bom
	stream.CopyTo(noBomStream);

	noBomStream.SaveToFile(filename, 2);                    // adSaveCreateOverWrite = 2
	noBomStream.Flush();
	noBomStream.Close();
}

stream.Close();

penpen

[dbenham]

Thanks penpen.

I don't need the BOM values, I just need to know that there is a BOM, and the size. It wasn't too hard to modify the strategy a bit.

I manage all ADO stream IO through a custom ADOStream object that emulates a TextStream object. I handle removal of the BOM in two steps.

First in the ADOStream constructor that opens the ADO stream, I added a NoBOM boolean parameter, defaulting to FALSE. The constructor determines if the stream is for output, and what the BOM length is, (if any) using the following:

Code: Select all

var bomSize = 0;
stream.Open();
if (mode !== _g.ForReading && noBom) {
  stream.WriteText("");
  stream.Position = bomSize = stream.Size;
}

Then my close routine that actually writes to file does some extra work if there is a BOM that needs to be removed:

Code: Select all

Close = function() {
  if (mode!==_g.ForReading){
    if (bomSize) {
      var noBomStream = WScript.CreateObject("ADODB.Stream");
      noBomStream.Type = 1;
      noBomStream.Mode = 3;
      noBomStream.Open();
      stream.Position = bomSize;
      stream.CopyTo(noBomStream);
      noBomStream.SaveToFile( name, 2 );
      noBomStream.Flush();
      noBomStream.Close();
      noBomStream = null;
    } else stream.SaveToFile( name, 2 );
  }
  stream.Close();
  stream=null;
}

The above works well, but I don't like the fact that the entire output must be stored in memory twice if the BOM is to be removed. I fear this could cause problems with large output files. But this seems to be the accepted norm for how to remove the BOM from ADO output.

I'm getting ready to post JREPL v7.10 with the ability to remove the BOM from ADO Unicode output.


Dave Benham

[dbenham]

Here is JREPL v7.10 with the new ability to remove the BOM from ADO Unicode output.

JREPL7.10.zip

Downloaded 94 times from the main release page in 12 days while v7.10 was the current release.

(26.57 KiB) Downloaded 906 times

The option to remove the BOM is specified by appending |NB to the /O outFile|CharSet parameter

For example /O "outFile.txt|utf-8|nb" would specify that the output should be written as UTF-8 without BOM.

Changed documentation excerpts:

Code: Select all

prompt> jrepl /?history

    2018-03-14 v7.10: Now can block BOM in ADO output files by appending |NB
                      to |CharSet in the /O option and OpenOutput() function.
<truncated>

Code: Select all

prompt> jrepl /?/o

      /O OutFile[|CharSet[|NB]]

            Output is written to file OutFile instead of stdout. Any existing
            OutFile is overwritten unless the /APP option is also used.

            If |CharSet (internet character set name) is appended to OutFile,
            then the file is opened via ADO using the specified CharSet value.
            The output line terminator still defaults to \r\n when using ADO,
            and may be changed to \n with the \U option. Both ADO and the
            CharSet must be available on the local system. Unicode files
            written by ADO have a BOM by default. Appending |NB (or |anyvalue)
            to the CharSet blocks the BOM from being written.

            If /F InFile is also used, then an OutFile value of "-" overwrites
            the original InFile with the output. A value of "-" preserves the
            original character set. A value of "-|" explicitly transforms the
            file into the machine default character set. A "-|CharSet" value
            explicitly transforms the file into the specified character set.
            The output is first written to a temporary file with the same path
            and name, with .new appended. Upon completion, the temp file is
            moved to replace the InFile.

            It is rarely useful, but /APP may be combined with /O -. But /APP
            cannot be combined with /O "-|CharSet".

Code: Select all

prompt> jrepl /?jscript
<truncated>
      openOutput( fileName[|CharSet[|NB]] [,appendBoolean [,utfBoolean]] )

               Open a new TextStream object for writing and assign it to the
               output variable. If appendBoolean is truthy, then open the file
               for appending.

               If |CharSet is appended to the fileName, then open the file
               using ADO and the specified internet character set name. The
               output variable will be set to an object that partially
               emulates a TextStream object (see the input object). Unicode
               written by ADO will have a BOM by default. The BOM is blocked
               by appending |NB (or |anyValue) to the CharSet.

               If utfBoolean is truthy, then output is encoded as unicode
               (UTF-16LE). The unicode file will automatically have the BOM
               unless opened for appending. The utfBoolean argument is ignored
               if |CharSet is also specified.

               If fileName is falsey, then output is written to stdout.

               All subsequent output will be written to the new destination.

               Any prior output file is automatically closed.
<truncated>               

Dave Benham

[dbenham]

When I added the |NB output option to suppress any BOM when writing ADO Unicode output, I forgot to provide a mechanism to suppress the BOM when using the /O - overwrite syntax.

Here is version 7.11 that adds the missing functionality to overwrite UTF without BOM:

JREPL7.11.zip

Downloaded 983 times from the main release page in a bit under 4 months while v7.11 was the current release.

(27.56 KiB) Downloaded 980 times

Here is an example usage that reads a file using the machine's native character set, changes all red into blue, and overwrites the file as UTF-8 without BOM.

Code: Select all

jrepl red blue /f file.txt /o "-|utf-8|nb"

Normally |NB is not used with the /I option (it is normally ignored when used with /I). But if the output is specified as /O - then it will not only preserve the character set used to read the file, but it will also use any |NB flag that may have been present on the input as an instruction to write the output without a BOM.

The following would simply remove any BOM from a UTF-8 file. If the BOM did not exist, then there would be no change:

Code: Select all

jrepl "^" "" /f "file.txt|utf-8|nb" /o -

Conversely, this example would add a BOM to a UTF-8 file if and only if it did not already have one. If the BOM already existed, then there would be no change.

Code: Select all

jrepl "^" "" /f "file.txt|utf-8" /o -

Dave Benham

[Mordru]

Hello, I've been using JREPL to encode/decode game data, (I've made an online batch game, but that's beside the point) and I have ran into a problem. People are able to intercept the calling parameters and therefore get my encode/decode table.

Is there any possible way I can copy/paste JREPL into my game code, and then call a subroutine for it within the batch file whenever I need to use it? That way people can't intercept the parameters when calling JREPL?