Alternate method to get TAB, Carriage return and possibly all others

[carlos]

It will very useful found a solution.

@carlos
I've found that, after issuing chcp command, initiating a new cmd instance on the same console would resolve the code page issue for that new cmd instance.
So for that particular version of genchr that I've used as test case in the previous post, putting chcp 437 in the start of the script would create correct chr files, because it offloads all of it's work to new instances of cmd on the same console. Otherwise the script needs to relaunch itself in a new instance of cmd after changing of the active code page.

Many thanks @sst. You found a solution.
I think the cmd relaunch is only needed for the copy command in text mode.

I check this code using the 950 codepage from the registry and now works ok with all the .chr files generated. Checked with this c code from 2014:
http://pastebin.com/raw/6K5SMYrg

I write your nick on the credits.

Edit: the codes is here: viewtopic.php?f=3&t=5326&p=56708#p56709

Is only a proposal of the new code. If you find a better way of fix that, please share the comments.

But I only test it on windows 10. Please if you have access to some of that windows versions, please you can do test? I want update the routine code for fix this bug (found 4 years after the creation). Genchr also have a page on ss64: https://ss64.com/nt/syntax-genchr.html Because that is needed update that with the fix

[sst]

I write your nick on the credits.

It's an honor for me and It's very kind of you carlos, but I prefer you not to. That was a great collaborative work that I had no role in creating of it. Recommendation to put a single chcp on the script does not make me eligible for the credit. But I appreciate your kindness, Really.
What is important here is the reason why this works for this version and not the others. That may comes in handy in future similar projects.

I think the cmd relaunch is only needed for the copy command in text mode.

My NLSAscii in it's core doesn't relay on copy but still needs relaunch until alternate solution can be found.

Is only a proposal of the new code. If you find a better way of fix that, please share the comments.

As for recommendations and comments, I think It should be continued on the main thread, so other readers who study the project can follow the progress. I will participate if any thing comes to my mind.

Cheers.

[carlos]

@sst seems that is needed relaunch the cmd instance as you says because cmd.exe use a internal variable CurrentCP that is only initialized when it start the first time, this variable is used on some internal methods, regarding of the call to chcp.exe, that calls to winapi functions SetConsoleOutputCP and SetConsoleCP that touch other variable.

The init of CurrentCP variables seems be like this:

If cmd not have a parent instance:
CurrentCP = get codepage from operating system locale or windows registry CodePage key.

If cmd have a parent instance:
CurrentCP = GetConsoleOutputCP()

The solution that you says is:
1) Change the current codepage with chcp
2) Run your batch code with cmd /c this will touch correctly the CurrentCP variable.

A possible workaround template:

Code: Select all

@echo off
::Workaround for environments with native dbcs codepages
if "%_cp_%"=="" (
  setlocal enableextensions
  for /f "tokens=*" %%a in ('chcp') do for %%b in (%%a) do set "_cp_=%%~nb"
  chcp 437 >nul
  cmd /d /c "%~f0"
  call chcp %%_cp_%% >nul
  exit /b
)

::your code here

[Aacini]

You don't need to rely on the internal format of any binary file. You may easily create your own binary file via CERTUTIL:

Code: Select all

@echo off
setlocal EnableDelayedExpansion

call :DefineCtrlChar BEL=07, TAB=09, LF=0A, CR=0D
echo BEL[%BEL%]BEL, LF[!LF!]LF, TAB[%TAB%]TAB, CR[!CR!]CR
goto :EOF


:DefineCtrlChar var=hex ...

> "CtrlChar.hex.txt" echo %2 20
certutil.exe -decodehex -f "CtrlChar.hex.txt" "CtrlChar.txt" > NUL
set /P "%1=" < CtrlChar.txt
set "%1=!%1:~0,1!"
shift & shift
if "%~2" neq "" goto DefineCtrlChar 
del "CtrlChar.*"
exit /B

Output:

Code: Select all

BEL[]BEL, LF[
]CR, TAB[       ]TAB, CR[

Antonio

[penpen]

i reread this topic, and thought about which codepages where part of windows over all versions:
I only found codepage 65000 (utf-7) and 65001 (utf-8).

Surprisingly using codepage transformation only suffices to solve this task (if i don't error);
Under windows xp this only works on patched cmd.exe (using carlos and Jason Hoods cmd.exe patch).
I actually have only access to win xp, 32 bit, german localization and windows 10, 64 bit, german localization, so i cannot test this anywhere else:

Code: Select all

@echo off
setlocal enableExtensions disableDelayedExpansion
:: set active, utf-7 and utf8 codepages
set /A "utf7Cp=65000, utf8Cp=65001"
set "cp="
:: thanks to carlos, for the chcp messages in other languages, see:
:: https://www.dostips.com/forum/viewtopic.php?f=3&t=8533#p56632
:: bugfixed: for /f "tokens=2 delims=:." %%a in ('2^>nul chcp') do set "cp=%%~a"
for /f "delims=." %%a in ('2^>nul chcp') do for %%b in (%%a) do set "cp=%%b"
if not defined cp for /f "tokens=2" %%a in ('2^>nul chcp') do set "cp=%%~a"
>nul chcp %utf8Cp%

:: define filenames
set "utf7DataFile=table.utf-7.txt"
set "dataFile=table.dat"

:: create chinese chcp seperator char
set "data="
<nul (>"%utf7DataFile%" set /p "=+/xo-")

for /f "tokens=2 delims=:." %%a in ("%cp%") do set "cp=%%~a"


:: create unicode data [U+1A00 : U+1AFF] using only characters in { '+', '-', '/', '0'-'9', 'A'-'Z', 'a'-'z' }
:: set "data="
:: set "data=%data%+GgAaARoCGgMaBBoFGgYaBxoIGgkaChoLGgwaDRoOGg8aEBoRGhIaExoUGhUaFhoXGhgaGRoaGhsaHBodGh4aHxogGiEaIhojGiQ"
:: set "data=%data%aJRomGicaKBopGioaKxosGi0aLhovGjAaMRoyGjMaNBo1GjYaNxo4GjkaOho7GjwaPRo+Gj8aQBpBGkIaQxpEGkUaRhpHGkgaSRp"
:: set "data=%data%KGksaTBpNGk4aTw-+GlAaURpSGlMaVBpVGlYaVxpYGlkaWhpbGlwaXRpeGl8aYBphGmIaYxpkGmUaZhpnGmgaaRpqGmsabBptGm4"
:: set "data=%data%abxpwGnEachpzGnQadRp2GncaeBp5Gnoaexp8Gn0afhp/GoAagRqCGoMahBqFGoYahxqIGokaihqLGowajRqOGo8akBqRGpIakxq"
:: set "data=%data%UGpUalhqXGpgamRqaGpsanBqdGp4anw-+GqAaoRqiGqMapBqlGqYapxqoGqkaqhqrGqwarRquGq8asBqxGrIasxq0GrUathq3Grg"
:: set "data=%data%auRq6GrsavBq9Gr4avxrAGsEawhrDGsQaxRrGGscayBrJGsoayxrMGs0azhrPGtAa0RrSGtMa1BrVGtYa1xrYGtka2hrbGtwa3Rr"
:: set "data=%data%eGt8a4BrhGuIa4xrkGuUa5hrnGuga6RrqGusa7BrtGu4a7w-+GvAa8RryGvMa9Br1GvYa9xr4Gvka+hr7Gvwa/Rr+-+Gv8-"
:: thanks to carlos, for finding a shorter data encoding, see:
:: https://www.dostips.com/forum/viewtopic.php?f=3&t=8513&p=56666#p56637
set "data="
set "data=%data%+GgAaARoCGgMaBBoFGgYaBxoIGgkaChoLGgwaDRoOGg8aEBoRGhIaExoUGhUaFhoXGhgaGRoaGhsaHBodGh4aHxogGiEaIhojGiQ"
set "data=%data%aJRomGicaKBopGioaKxosGi0aLhovGjAaMRoyGjMaNBo1GjYaNxo4GjkaOho7GjwaPRo+Gj8aQBpBGkIaQxpEGkUaRhpHGkgaSRp"
set "data=%data%KGksaTBpNGk4aTxpQGlEaUhpTGlQaVRpWGlcaWBpZGloaWxpcGl0aXhpfGmAaYRpiGmMaZBplGmYaZxpoGmkaahprGmwabRpuGm8"
set "data=%data%acBpxGnIacxp0GnUadhp3GngaeRp6GnsafBp9Gn4afxqAGoEaghqDGoQahRqGGocaiBqJGooaixqMGo0ajhqPGpAakRqSGpMalBq"
set "data=%data%VGpYalxqYGpkamhqbGpwanRqeGp8aoBqhGqIaoxqkGqUaphqnGqgaqRqqGqsarBqtGq4arxqwGrEashqzGrQatRq2GrcauBq5Gro"
set "data=%data%auxq8Gr0avhq/GsAawRrCGsMaxBrFGsYaxxrIGskayhrLGswazRrOGs8a0BrRGtIa0xrUGtUa1hrXGtga2RraGtsa3BrdGt4a3xr"
set "data=%data%gGuEa4hrjGuQa5RrmGuca6BrpGuoa6xrsGu0a7hrvGvAa8RryGvMa9Br1GvYa9xr4Gvka+hr7Gvwa/Rr+Gv8-"
::  index       0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
::              0         1         2         3         4         5         6         7         8         9         
<nul (>"%utf7DataFile%" set /p "=%data%")
:: ensure cmd.exe is in ANSI mode (file size should be 695 bytes; if it is 1390 then cmd.exe is in UCS-2 mode)
:: for %%a in ("%utf7DataFile%") do (if %%~za == 1390 (cmd /d /a /e:on /v:off /c"%~f0" & goto :eof))
:: size of data changed  to 685 bytes, so have to check for 1370
for %%a in ("%utf7DataFile%") do (if %%~za == 1370 (cmd /d /a /e:on /v:off /c"%~f0" & goto :eof))
>nul chcp %utf7Cp%
set "data=" & <"%utf7DataFile%" set /p "data="
>nul chcp %cp%
rem del "%utf7DataFile%"

:: create *.chr files
if not exist chars md chars
set "hex=0 1 2 3 4 5 6 7 8 9 A B C D E F"
set "index=0"
>nul chcp %utf8Cp%
cmd /d /u /e:on /v:on /c"@for %%h in (%hex%) do @for %%l in (%hex%) do @((for %%a in ("!index!") do @(>"%dataFile%" echo(!data:~0x%%~h%%~l,2!))&(>nul copy /a "%dataFile%" "chars\%%~h%%~l.chr" /b))"
>nul chcp %cp%
REM create "chars\1A.chr" because it's the eof character
>nul copy /a nul "chars\1A.chr"

:: build test file [0x00 : 0xFF]
for %%h in (%hex%) do @for %%l in (%hex%) do for %%a in ("chars\%%~h%%~l.chr") do if not "%%~za" == "1" echo(Error in file "%%~a".
if exist "%dataFile%" del "%dataFile%"
>nul (
	copy /b nul "%dataFile%"
	for %%h in (%hex%) do @for %%l in (%hex%) do copy /b "%dataFile%" +"chars\%%~h%%~l.chr" "%dataFile%"
)
goto :eof

penpen

Edit: Fixed a bug (set cp for different languages see viewtopic.php?f=3&t=8533#p56632).
Edit2: Applied the shorter data encoding, proposed by carlos here.

[carlos]

@penpen: nice technique, is new for me. Please you can explain it a little bit.?
Also, If I remember well, the utf-8 bug on windows xp only affects when you run code in a batch file, because cmd tries convert the source code of the batch file to that codepage calling to a winapi function with a incorrect parameter. But you can work on interactive mode using utf-8 If I remember well, thus if you convert your code snippet to macro, maybe you can run that code if you run it on interactive mode. Maybe: echo %macro% | cmd
Is a theory.

@aacini: yes, with certutil you can easily handle binary data, but the idea i think is develop a new technique, as alternative in case certutil change or is removed in the future in the default installation, as it happened to the old telnet.exe

[penpen]

Well the basic idea is the same as used in the "genchr.cmd":
Create files with the searched byte (0xst; s,t in {0..F}) and an eof character (0x1A) at position2 and then use copy to cut of eof character and following data.

I only used a way to do that using utf-7 and utf-8 codepages.
So instead of using "makecab.exe" to create those bytes, i interpreted those two bytes as an ucs-2 encoded character.
Because the utf-16le characters in this range are identical, i needed an utf-16le file that contains all unicode characters from U+1A00 to U+1AFF (0xst1A == U+1Ast).

I used type to convert that file to utf-7 encoding:

Code: Select all

Z:\>chcp 65000
Aktive Codepage: 65000.

Z:\>>"utf7.txt" type "utf16le.txt"

Z:\>

The code simply reverses this step, by creating the utf-7 file, changing the codepage to utf-7 and use set/p to read the file.
Then i used "cmd/u" to create the single character files containing the needed utf-16le character.

I then tested if the utf-8 codepage is sufficient for cutting off the eof/sub character, and it seems it works (but i cannot be sure for other versions/localizations of windows):
That was a bit lucky, because some bytes are also leading bytes in multibyte encoding of characters (for example 0xC2);
i assume that 0x1A never is a trailing byte saved this solution.


penpen

[sst]

@Aacini
The aim was to cover pre vista as well. cerutil was available by default starting from vista if I'm not wrong. On XP we need to install Server 2003 Admin Pack and I think its not available for Win2K at all.

@penpen
Very nice and clever approach. It is very unfortunate that it cannot be used strait on XP.
I didn't get purpose for the index variable and extra FOR loop around it, may its a leftover or something. I'm not sure.
Anyway I got the concept, Thanks for that.

[carlos]

@penpen, now I see that the core of the function is rely on a internal conversion from utf-7 to utf-16le using set /p.
I test on windows 10 ok, with some changes, I use codepage 437 instead utf-8 and works ok, also I encode the utf16 file to utf-7 with convertcp and the result encoded is a little more small. It also works.

Code: Select all

set "data="
set "data=%data%+GgAaARoCGgMaBBoFGgYaBxoIGgkaChoLGgwaDRoOGg8aEBoRGhIaExoUGhUaFhoXGhgaGRoaGhsaHBodGh4aHxogGiEaIhojGiQ"
set "data=%data%aJRomGicaKBopGioaKxosGi0aLhovGjAaMRoyGjMaNBo1GjYaNxo4GjkaOho7GjwaPRo+Gj8aQBpBGkIaQxpEGkUaRhpHGkgaSRp"
set "data=%data%KGksaTBpNGk4aTxpQGlEaUhpTGlQaVRpWGlcaWBpZGloaWxpcGl0aXhpfGmAaYRpiGmMaZBplGmYaZxpoGmkaahprGmwabRpuGm8"
set "data=%data%acBpxGnIacxp0GnUadhp3GngaeRp6GnsafBp9Gn4afxqAGoEaghqDGoQahRqGGocaiBqJGooaixqMGo0ajhqPGpAakRqSGpMalBq"
set "data=%data%VGpYalxqYGpkamhqbGpwanRqeGp8aoBqhGqIaoxqkGqUaphqnGqgaqRqqGqsarBqtGq4arxqwGrEashqzGrQatRq2GrcauBq5Gro"
set "data=%data%auxq8Gr0avhq/GsAawRrCGsMaxBrFGsYaxxrIGskayhrLGswazRrOGs8a0BrRGtIa0xrUGtUa1hrXGtga2RraGtsa3BrdGt4a3xr"
set "data=%data%gGuEa4hrjGuQa5RrmGuca6BrpGuoa6xrsGu0a7hrvGvAa8RryGvMa9Br1GvYa9xr4Gvka+hr7Gvwa/Rr+Gv8-"

Unfortunately, this nice new technique will not works on cmd for windows xp, unless apply the unofficial mentioned patch, because that bug also affects utf-7, thus without that patch, set /p cannot convert the utf-7 to utf-16 properly on windows xp.
But, I think that support for xp is not really needed. This new technique is a nice alternative.

[penpen]

I use codepage 437 instead utf-8 and works ok

Does that mean the batch produces errors when using utf-8, or what is the reason to not use it?
If the answer is yes, in what error occurs in which circumstances (which initial codepage; is cmd.exe in ANSI mode (<-cmd/a) or in UCS-2 mode (<-cmd/u))?

What i fear most is, that copy might use a fallback codepage, if reading an invalid character in utf-8:
I hope that didn't happen.

This new technique is a nice alternative.

Many thanks.

Also thanks for the chcp messages in other languages;
i've bugfixed the above code with an own (only slightly different) version, that could be extended easily to handle additional ending characters in other languages (although i don't know if there are any).


penpen

[carlos]

I replace utf-8 by 437 trying avoid the utf-8 bug on xp, but finally the bug also extend to utf-7. If you look the unofficial patch source it covers utf-8 and utf-7. I got is a nice technique and a real replacement for certutil. For xp and windows 2000 we always can use debug.exe for generate binary data (i have a old program that generates scripts). We can also create the patched cmd.exe using a debug.exe script. I not know why I not create the unofficial patch for cmd.exe using a debug script. I will have fun writing that. In that case if you want support xp you call the debug script for patch the cmd. That would be nice.

[sst]

This is the improved version of NLSAscii.cmd
Additions/Changes:
- It now relaunches itself to properly run under the correct code page.
- Integrity verification code added to ASCII generator function so it can now reliably determine if the correct table has been produced.
- Greatly improved the performance of generator by some small changes. On my system the execution time reduced from around 0.35 seconds without integrity verification to around 0.12 seconds with integrity verification and 0.06 seconds without integrity check.

Please note this not a full batch application,it is only a demo to show how to handle the NLS files and extract a complete ASCII table from them. However incorporating the code for your own use is straightforward.

Code: Select all

:: NLSAscii.cmd

:: NLS ASCII TABLE Generator. By sst
:: Special thanks to carlos for his valuable feedback and guidances

:: Sample batch script to demonstrate creation of in-memory ASCII table using NLS files.
:: Dependency:
:: C_950.NLS  SHA1: 1cd3f1ccf03d2b2d00dd3a133a6bebaa0e1bdb89
:: C_1252.NLS SHA1: 355e2ada0b9ea4f2a844c2d236d1b48336881b22

:: This script should produce the correct table on Win2K, WinXP, WinVista, Win7, Win8.x, Win10

:: Tested on Win2K SP4, WinXP SP3 32bit, Win7 SP1 64bit, Win10 1709 64bit




@echo off
setlocal EnableExtensions DisableDelayedExpansion

:: Using a unique enough name for init variable to effectively reduce the chance of unintentional collisions to zero.
set "Init=NLSAscii.Init.{10355d30-4e90-11e8-9330-aa819362ab44}"
if not defined %Init% (
    %= Relaunching in new cmd instance to fully apply new code page. =%
    setlocal EnableDelayedExpansion
    set "errorlevel="
    set "chcp=chcp"
    >nul 2>&1 !chcp!
    if errorlevel 1 (
        set "errorlevel=!errorlevel!
        echo,
        <nul set /p "=WARNING: "
        if "!errorlevel!"=="9009" (echo CHCP not found.) else (echo CHCP Error.)
        echo Correct operation of program depends on the current console code page.
        set "chcp=REM,"
        set "errorlevel="
    )
    for /f "tokens=*" %%a in ('!chcp!') do for %%b in (%%a) do set "PreviousCodePage=%%~nb"
    >nul !chcp! 437
    set "%Init%=1"
    "%COMSPEC%" /a /d /c "%~f0"
    set "errorlevel=!errorlevel!
    >nul 2>&1 !chcp! !PreviousCodePage! || echo WARNING: Could not revert back to the initial code page !PreviousCodePage!
    exit /b !errorlevel!
)
set "%Init%="
set "Init="
set "TIME="

:: Relaunched in new cmd instance, actual code begins here

:: Set this to none zero to enable creation of one-byte .chr files.
set /a "EnableChrExport=1"
set "ChrSubDir=.\CHR_NLS"

echo,
<nul set /p "=Generating and verfying ASCII table ... "
set "Start=%TIME%"
call :CreateAsciiTableFromNLS ASCII_TABLE ErrMsg
if errorlevel 1 (
    echo Failed.
    (echo,%ErrMsg%)>&2
    echo,
    if /i %errorlevel% EQU 0x7FFFFFFF (
        echo Integrity verfication failed.
        echo This can happen for the following reasons:
        echo     - The batch file is running with an inappropriate code page,
        echo       The default working code page should be 437
        echo     - The NLS files [C_1252.NLS/C_950.NLS] may either be corrupt,
        echo       or have different binary structure than what is expected.
        echo,
    )
    pause
    exit /b %errorlevel%
)
set "End=%TIME%"
call :timediff Elapsed Start End
echo OK.   Time taken: %Elapsed%

setlocal EnableDelayedExpansion
<nul>"ASCII_TABLE.dat" set /p "=!ASCII_TABLE!"

echo,
echo Ascii table has been created in ASCII_TABLE environment variable.
echo,
echo For manual verification of the generated table, The contents of ASCII_TABLE variable
echo have been written to the file "ASCII_TABLE.dat" in the current directory.
echo SHA1 hash sum of the file should be: 8760d3807fb0c8ce8fd426f4cf72632edf8789f6
echo,

set "ASCII_TABLE=#!ASCII_TABLE!"
set "LF=!ASCII_TABLE:~10,1!"
set "CR=!ASCII_TABLE:~13,1!"
set "SUB=!ASCII_TABLE:~26,1!"
if /i %EnableChrExport% NEQ 0 (
    <nul set /p "=Do you want to additionally create one-byte .chr files?[YN] "
    set "Key="
    for /F "delims=" %%K in ('"xcopy ?^| /w 2>nul"') do if not defined Key set "Key=%%K"
    if /i "!Key:~-1!"=="Y" (set "Key=Y") else (set "Key=N")
    echo !Key!
    if "!Key!"=="Y" (
        <nul set /p "=Please Wait...!CR!"
        if not exist "%ChrSubDir%\" MD "%ChrSubDir%"
        set "DEC=%ChrSubDir%\DEC"
        if not exist "!DEC!\" MD "!DEC!"
        for /L %%i in (1,1,255) do (
           (echo !ASCII_TABLE:~%%i,1!%SUB%)>"!DEC!\%%i.tmp"
           copy /y "!DEC!\%%i.tmp" /a "!DEC!\%%i.chr" /b >nul
        )
        set "prompt=%SUB%"
        "%COMSPEC%" /a /d /k<nul>"!DEC!\26.chr"
        "%COMSPEC%" /u /d /k<nul>"!DEC!\0.tmp"
        "%COMSPEC%" /a /d /k<nul>>"!DEC!\0.tmp"
        type "!DEC!\0.tmp"|(pause>nul&findstr "^")>"!DEC!\0.tmp"
        copy /y "!DEC!\0.tmp" /a "!DEC!\0.chr" /b >nul
        del /f /q "!DEC!\*.tmp" >nul 2>&1

        set "HEX=%ChrSubDir%\HEX"
        if not exist "!HEX!\" MD "!HEX!"
        set "HD=0 1 2 3 4 5 6 7 8 9 A B C D E F"
        set "i=0"
        for %%H in (!HD!) do for %%L in (!HD!) do (
            copy /y "!DEC!\!i!.chr" "!HEX!\%%H%%L.chr" >nul
            set /a "i+=1"
        )

        echo Done.          !LF!
    )
)
endlocal



pause
exit /b 0



:CreateAsciiTableFromNLS <outResult> [outErrorMessage]
setlocal EnableDelayedExpansion

set "NLS950=%SystemRoot%\system32\C_950.NLS"
set "NLS1252=%SystemRoot%\system32\C_1252.NLS"

if "%~1"=="" exit /b 2
if not exist "%NLS950%"  (endlocal & 2>nul set "%~2=%0 : "%NLS950%" does not exist." & exit /b 1)
if not exist "%NLS1252%" (endlocal & 2>nul set "%~2=%0 : "%NLS1252%" does not exist." & exit /b 1)

:: Detection for Win2K is just for speed of reading file content
:: On Win2K, "SET /P" will read 1024 bytes of input buffer
:: On WinXP and later "SET /P" will read 1023 bytes of input buffer.
:: We need to know this to be able to seek to the correct offset.
:: Otherwise we have to use exsclusively "pause>nul" to seek the input file
:: which is much slower for larger offsets
:: PS Note:
::   There is a difference between using SET /P inside a pipe eg: TYPE "BinaryFile" | SET /P "CAPTURE="
::   and using SET /P to directly read file contents eg: (SET /P "CAPTURE=")<"BinaryFile"
::   In pipe mode "SET /P" will always eat the first 1023 bytes of input (1024 bytes on Win2K), no matter what the content is
::   and the result will be truncated at <NULL>, <CRLF> and <LFCR>
::   In direct read mode, SET /P will eat the whole 1023 or 1024 bytes ONLY if doesn't encounter one of the <CRLF> or <LFCR> pairs in the way,
::   in that case, SET /P will immediately stop reading from input upon reaching <CRLF> or <LFCR>
::   In both modes, all of the control characters(range 00-1F) which came immediately before <CRLF> or <LFCR> will be removed from the target variable.
::   Here, using the direct read mode to fast seek the NLS files is OK becuase there is no <CRLF> or <LFCR> in them.
::   But for the perpose of FastSeeking arbitrary files, only pipe mode can be used.

for /F "tokens=3" %%V in ('ver') do (
    if "%%V"=="2000" (set "FastSeekBytes=1024") else (set "FastSeekBytes=1023")
)
set "@{FastSeek}=SET /P "=""

:: Win2K's cmd has a bug in which it expands variable values to zero in SET /A expressions,
:: if they are placed in the right side of the operands. Thus we have to expand them ourselves.
set /a "NLS950_SingleSeekBytes1=1782-%FastSeekBytes%"
set /a "NLS950_SingleSeekBytes2=1890-%FastSeekBytes%"
set /a "NLS1252_SingleSeekBytes=547"

:: Capture phase
(
    for /L %%P in (1,1,%NLS1252_SingleSeekBytes%) do pause
    set /p "CAPTURE="
    set "CHAR_01-7F=!CAPTURE:~0,127!"
    set "CHAR_A0-FF=!CAPTURE:~159,96!"
    set "CHAR_EF=!CHAR_A0-FF:~79,1!"
)<"%NLS1252%">nul
(
    %@{FastSeek}%
    for /L %%P in (1,1,%NLS950_SingleSeekBytes1%) do pause
    REM Equivalent to: for /L %%P in (1,1,1782) do pause>nul

    set /p "CAPTURE="
    set "CHAR_80-93=!CAPTURE:%CHAR_EF%=!
    set "CHAR_80-93=!CHAR_80-93:~0,20!
)<"%NLS950%">nul
(
    %@{FastSeek}%
    for /L %%P in (1,1,%NLS950_SingleSeekBytes2%) do pause
    REM Equivalent to: for /L %%P in (1,1,1890) do pause>nul

    set /p "CAPTURE="
    set "CHAR_94-9F=!CAPTURE:%CHAR_EF%=!
    set "CHAR_94-9F=!CHAR_94-9F:~0,12!
)<"%NLS950%">nul

:: Verification phase
set "CAPTURE="
set "TIME="
set "RANDOM="
set /a "_ErrLevel=0x7FFFFFFF"
set "CBool=^!^!"
set "BasePath=."
if defined Temp if exist "%Temp%\" set "BasePath=%Temp%"
for /F "tokens=1,3 delims=0123456789" %%A in ("%TIME: =0%") do set "time.delims=%%A%%B"
for /F "tokens=1-4 delims=%time.delims%" %%A in ("%TIME: =0%") do set "unq=%%A%%B%%C%%D"
set "unq=TMPNLSASCVRFY-%unq:~0,8%-%RANDOM%
set "Verify.File=%BasePath%\_asciitable_%unq%.tmp"
set "Dumper.File=%BasePath%\_dumper_%unq%.tmp"
<nul>"%Verify.File%" set /p "=!CHAR_01-7F!!CHAR_80-93!!CHAR_94-9F!!CHAR_A0-FF!"
set "CHAR_01-7F="&set "CHAR_80-93="&set "CHAR_94-9F="&set "CHAR_A0-FF="&set "unq="
set "a32=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
set "Dumper.Data=%a32%%a32%%a32%#%a32%%a32%%a32%%a32%%a32:~2%" =96xa + 1x# + 158xa=255
for %%F in ("%Verify.File%") do if "%%~zF"=="255" (
    <nul>"%Dumper.File%" set /p "=%Dumper.Data%"
    set "a32="&set "Dumper.Data="
    set /a "index=cmp=0"
    for /F "skip=1 tokens=2" %%i in ('fc /b "%Verify.File%" "%Dumper.File%"') do (
        if "!cmp!"=="0" (
            set /a "index+=1, cmp=index-0x%%i" 2>nul || set "cmp=1"
        )
    )
    set /a "_ErrLevel*=!CBool!(cmp|(255-index))"
)
(
    endlocal
    if /i %_ErrLevel% EQU 0 (
        <"%Verify.File%">nul set /p "%~1="
    ) else (
        2>nul set "%~2=%0 : ASCII table error, Integrity verfication failed."
    )
    del /f "%Verify.File%", "%Dumper.File%" >nul 2>&1
    exit /b %_ErrLevel%
)



:: timediff
:: -> Input and output format is compatible with %TIME%
:: -> The time and decimal seperators can be different between StartTime and EndTime
:: -> Output will have the same time and decimal seperators as returned by %TIME% at runtime.
:: -> If EndTime is less than StartTime then:
::     EndTime will be treated as a time in the next day
::     in that case, function measures time difference between a maximum distance of 24 hours minus 1 centisecond
::     time elements can have values greater than their standard maximum value ex: 12:247:853.5214
::     provided than the total represented time does not exceed 24*360000 centiseconds (24:00:00.00)
::     otherwise the result will not be meaningful.
:: -> If EndTime is greater than or equals to StartTime then:
::     No formal limitation applies to the value of elements,
::     except that total represented time can not exceed 2147483647 centiseconds.
:: -> To normalize (convert) none-standard time values pass 0:0:0.0 as StartTime
:timediff <outDiff> <inStartTime> <inEndTime>
(
    setlocal EnableDelayedExpansion
    set "TIME=" & set "Input=!%~2!#!%~3!"
    for /F "tokens=1,3,4,5,7,9,11 delims=0123456789 " %%A in ("!Input!#!TIME!") do (
        set "user.time.delims=%%A%%B%%C%%D%%E " & set "sys.time.delims=%%F%%G"
    )
)
for /F "tokens=1-8 delims=%user.time.delims%" %%a in ("%Input%") do (
    for %%A in ("@h1=%%a" "@m1=%%b" "@s1=%%c" "@c1=%%d" "@h2=%%e" "@m2=%%f" "@s2=%%g" "@c2=%%h") do (
        for /F "tokens=1,2 delims==" %%A in ("%%~A") do (
            for /F "tokens=* delims=0" %%B in ("%%B") do set "%%A=%%B"
        )
    )
    set /a "@d=(@h2-@h1)*360000+(@m2-@m1)*6000+(@s2-@s1)*100+(@c2-@c1), @sign=(@d>>31)&1, @d+=(@sign*24*360000), @h=(@d/360000), @d%%=360000, @m=@d/6000, @d%%=6000, @s=@d/100, @c=@d%%100"
)
(
    if /i %@h% LEQ 9 set "@h=0%@h%"
    if /i %@m% LEQ 9 set "@m=0%@m%"
    if /i %@s% LEQ 9 set "@s=0%@s%"
    if /i %@c% LEQ 9 set "@c=0%@c%"
)
(
    endlocal
    set "%~1=%@h%%sys.time.delims:~0,1%%@m%%sys.time.delims:~0,1%%@s%%sys.time.delims:~1,1%%@c%"
    exit /b
)

[penpen]

We can also create the patched cmd.exe using a debug.exe script. I not know why I not create the unofficial patch for cmd.exe using a debug script. I will have fun writing that. In that case if you want support xp you call the debug script for patch the cmd. That would be nice.

That should be no problem.

Although in case of xp:
I'm unsure why you don't create the character files using a debug script, or
you don't use the code you gave here.

Sorry for the delay, our public utility (don't know if that is the proper translation of "Stadtwerke") are checking and replacing all power cables in our street if necessary... (probably takes additional 1-2 weeks...).

penpen

[penpen]

We can also create the patched cmd.exe using a debug.exe script. I not know why I not create the unofficial patch for cmd.exe using a debug script. I will have fun writing that. In that case if you want support xp you call the debug script for patch the cmd. That would be nice.

That should be no problem.

Although in case of xp:
I'm unsure why you don't create the character files using a debug script, or
why you don't use the code you gave here.

Sorry for the delay, our public utility (don't know if that is the proper translation of "Stadtwerke") are checking and replacing all power cables in our street if necessary... (probably takes additional 1-2 weeks...).

penpen

[sst]

I was thinking about 8191 characters limit of cmd, and though what will happen if I echo a string that is exactly 8191 characters long, what will happen to the trailing CRLF that echo appends to end of the string? luckily with delayed expansion its possible to echo such a string and the length of the command will not limit the characters that can be passed to echo so I did this

Code: Select all

echo !String8191!
echo Hello

And as I guessed, echo was unable to append CRLF at the end of String8191 so "Hello" appeared on the same line. Surely this is not new but I thought that if I eliminate one character from the string and reduce it to 8190 characters then there will be room only for the CR to be appended by echo and the obtrusive LF will go to hell. Now I can catch her alone

Code: Select all

@echo off
setlocal DisableDelayedExpansion

for /F %%Z in ('
    @"CMD " /e:on /v:on /q /d /c
    set "line= "^^^&
    (for /L %%Z in (1 1 12^) do set "line=!line!!line!"^)^^^&
    echo(!line!!line:~0^,-2!^^^&echo(
') do set "CR=%%Z"

setlocal EnableDelayedExpansion
echo(      World!CR!Hello&
pause

This is ugly and crazy and probably have no real use, but the hunting was interesting for me so I shared the story.