new functions: :chr, :asc, :asciiMap

[dbenham]

Here is code I developed for 3 new functions to support bi-directional conversion between numeric ASCII codes and characters. I think I saw routines with similar functionality some where that relied on DEBUG and temp files. These routines do not need either.

There are 10 characters that are problematic when trying to put them in a variable that are not supported. Jeb has some tricks for some (all?) of them. But I'm not sure it would be productive for this application.

code...

Code: Select all

@echo off
setlocal disableDelayedExpansion
for /l %%n in (0,1,255) do (
  call :chr %%n char
  if not errorlevel 1 (
    call :asc char 0 n
    call echo "%%n:%%char%%:%%n%%"
  ) else echo chr %%n produced above error
)
exit /b

:asc        StrVar IntVal [RtnVar]
::
::  Computes the ASCII code for a specified character within the string
::  contained by variable StrVar. The position within the string is specified
::  by the IntVal argument. A non-negative value is relative to the beginning
::  of the string, with 0 specifying the first character. A negative value is
::  relative to the end of the string, with -1 specifying the last character.
::
::  Sets RtnVar=result
::  or displays result if RtnVar not specified
::
::  IntVal may be passed as a variable without enclosing the name in quotes.
:::
::: Dependencies - asciiMap
:::
  setlocal disableDelayedExpansion
  set /a n=%~2 2>nul
  call set "chr=%%%~1:~%n%,1%%"
  call :asciiMap ascii
  setlocal enableDelayedExpansion
  if "!chr!"==" " set /a rtn=32&goto :asc.end
  set rtn=
  for /l %%n in (0,1,255) do if "!ascii:~%%n,1!"=="!chr!" set rtn=%%n
  :asc.end
  (endlocal & rem -- return values
    endlocal
    if "%~3" neq "" (set %~3=%rtn%) else (echo:%rtn%)
  )
exit /b


:chr        IntVal [RtnVar]
::
::  Converts ASCII code IntVal into the corresponding character.
::
::  Sets RtnVar=result
::  or displays result if RtnVar not specified
::
::  IntVal must be a value between 0 and 255.
::
::  Aborts with an error message to stderr and errorlevel 1 if IntVal
::  corresponds to one of the following problematic characters:
::
::    Dec   Hex   Oct  Char
::    ---  ----  ----  ----
::      0  0x00    00  NUL  (null)
::      3  0x03    03  ETX  (end of text)
::      8  0x08   010  BS   (backspace)
::      9  0x09   011  TAB  (horizontal tab)
::     10  0x0A   012  LF   (line feed)
::     11  0x0B   013  VT   (vertical tab)
::     12  0x0C   014  FF   (form feed)
::     13  0x0D   015  CR   (carriage return)
::     26  0x1A   032  SUB  (substitute)
::    127  0x7F  0177  DEL  (delete)
::
::  IntVal may be passed as a variable without enclosing the name in quotes.
:::
::: Dependencies - asciiMap
:::
  setlocal disableDelayedExpansion
  set /a n=%~1 2>nul
  if %n%==32 set "c= "&goto :chr.end
  call :asciiMap map
  call set "c=%%map:~%n%,1%%"
  if "%c%%c%"=="  " (
    echo ERROR: Problematic ASCII Code >&2
    exit /b 1
  )
  :chr.end
  (endlocal & rem -- return values
    if "%~2" neq "" (set %~2=^%c%) else (echo:^%c%)
  )
exit /b


:asciiMap   rtnVar
::
::  Sets variable rtnVar to a 256 character string containing the complete
::  extended ASCII character set except a space has been substituted for each
::  of the following problematic characters:
::
::    Dec   Hex   Oct  Char
::    ---  ----  ----  ----
::      0  0x00    00  NUL  (null)
::      3  0x03    03  ETX  (end of text)
::      8  0x08   010  BS   (backspace)
::      9  0x09   011  TAB  (horizontal tab)
::     10  0x0A   012  LF   (line feed)
::     11  0x0B   013  VT   (vertical tab)
::     12  0x0C   014  FF   (form feed)
::     13  0x0D   015  CR   (carriage return)
::     26  0x0A   032  SUB  (substitute)
::    127  0x7F  0177  DEL  (delete)
:::
::: Dependencies - <none>
:::
  set %~1= 
         !^"#$%%^&'^(^)*+,-./0123456789:;^<=^>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^^_`abcdefghijklmnopqrstuvwxyz{^|}~ €‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ
exit /b

and output...

Code: Select all

ERROR: Problematic ASCII Code 
chr 0 produced above error
"1:
:1"
"2::2"
ERROR: Problematic ASCII Code 
chr 3 produced above error
"4::4"
"5::5"
"6::6"
"7::7"
ERROR: Problematic ASCII Code 
chr 8 produced above error
ERROR: Problematic ASCII Code 
chr 9 produced above error
ERROR: Problematic ASCII Code 
chr 10 produced above error
ERROR: Problematic ASCII Code 
chr 11 produced above error
ERROR: Problematic ASCII Code 
chr 12 produced above error
ERROR: Problematic ASCII Code 
chr 13 produced above error
"14::14"
"15::15"
"16::16"
"17::17"
"18::18"
"19::19"
"20::20"
"21::21"
"22::22"
"23::23"
"24::24"
"25::25"
ERROR: Problematic ASCII Code 
chr 26 produced above error
"27::27"
"28::28"
"29::29"
"30::30"
"31::31"
"32: :32"
"33:!:33"
"34:":34"
"35:#:35"
"36:$:36"
"37:%:37"
"38:&:38"
"39:':39"
"40:(:40"
"41:):41"
"42:*:42"
"43:+:43"
"44:,:44"
"45:-:45"
"46:.:46"
"47:/:47"
"48:0:48"
"49:1:49"
"50:2:50"
"51:3:51"
"52:4:52"
"53:5:53"
"54:6:54"
"55:7:55"
"56:8:56"
"57:9:57"
"58:::58"
"59:;:59"
"60:<:60"
"61:=:61"
"62:>:62"
"63:?:63"
"64:@:64"
"65:A:65"
"66:B:66"
"67:C:67"
"68:D:68"
"69:E:69"
"70:F:70"
"71:G:71"
"72:H:72"
"73:I:73"
"74:J:74"
"75:K:75"
"76:L:76"
"77:M:77"
"78:N:78"
"79:O:79"
"80:P:80"
"81:Q:81"
"82:R:82"
"83:S:83"
"84:T:84"
"85:U:85"
"86:V:86"
"87:W:87"
"88:X:88"
"89:Y:89"
"90:Z:90"
"91:[:91"
"92:\:92"
"93:]:93"
"94:^:94"
"95:_:95"
"96:`:96"
"97:a:97"
"98:b:98"
"99:c:99"
"100:d:100"
"101:e:101"
"102:f:102"
"103:g:103"
"104:h:104"
"105:i:105"
"106:j:106"
"107:k:107"
"108:l:108"
"109:m:109"
"110:n:110"
"111:o:111"
"112:p:112"
"113:q:113"
"114:r:114"
"115:s:115"
"116:t:116"
"117:u:117"
"118:v:118"
"119:w:119"
"120:x:120"
"121:y:121"
"122:z:122"
"123:{:123"
"124:|:124"
"125:}:125"
"126:~:126"
ERROR: Problematic ASCII Code 
chr 127 produced above error
"128:€:128"
"129::129"
"130:‚:130"
"131:ƒ:131"
"132:„:132"
"133:…:133"
"134:†:134"
"135:‡:135"
"136:ˆ:136"
"137:‰:137"
"138:Š:138"
"139:‹:139"
"140:Œ:140"
"141::141"
"142:Ž:142"
"143::143"
"144::144"
"145:‘:145"
"146:’:146"
"147:“:147"
"148:”:148"
"149:•:149"
"150:–:150"
"151:—:151"
"152:˜:152"
"153:™:153"
"154:š:154"
"155:›:155"
"156:œ:156"
"157::157"
"158:ž:158"
"159:Ÿ:159"
"160: :160"
"161:¡:161"
"162:¢:162"
"163:£:163"
"164:¤:164"
"165:¥:165"
"166:¦:166"
"167:§:167"
"168:¨:168"
"169:©:169"
"170:ª:170"
"171:«:171"
"172:¬:172"
"173:­:173"
"174:®:174"
"175:¯:175"
"176:°:176"
"177:±:177"
"178:²:178"
"179:³:179"
"180:´:180"
"181:µ:181"
"182:¶:182"
"183:·:183"
"184:¸:184"
"185:¹:185"
"186:º:186"
"187:»:187"
"188:¼:188"
"189:½:189"
"190:¾:190"
"191:¿:191"
"192:À:192"
"193:Á:193"
"194:Â:194"
"195:Ã:195"
"196:Ä:196"
"197:Å:197"
"198:Æ:198"
"199:Ç:199"
"200:È:200"
"201:É:201"
"202:Ê:202"
"203:Ë:203"
"204:Ì:204"
"205:Í:205"
"206:Î:206"
"207:Ï:207"
"208:Ð:208"
"209:Ñ:209"
"210:Ò:210"
"211:Ó:211"
"212:Ô:212"
"213:Õ:213"
"214:Ö:214"
"215:×:215"
"216:Ø:216"
"217:Ù:217"
"218:Ú:218"
"219:Û:219"
"220:Ü:220"
"221:Ý:221"
"222:Þ:222"
"223:ß:223"
"224:à:224"
"225:á:225"
"226:â:226"
"227:ã:227"
"228:ä:228"
"229:å:229"
"230:æ:230"
"231:ç:231"
"232:è:232"
"233:é:233"
"234:ê:234"
"235:ë:235"
"236:ì:236"
"237:í:237"
"238:î:238"
"239:ï:239"
"240:ð:240"
"241:ñ:241"
"242:ò:242"
"243:ó:243"
"244:ô:244"
"245:õ:245"
"246:ö:246"
"247:÷:247"
"248:ø:248"
"249:ù:249"
"250:ú:250"
"251:û:251"
"252:ü:252"
"253:ý:253"
"254:þ:254"
"255:ÿ:255"

Dave Benham

[jeb]

Nice,
but you should be able to output each character, only the NUL=0=0x00 character should be hard.
I never found a way to handle it in a variable.

The problematic characters can be handeld with delayed expansion.
But <CR> and <LF> can be inserted in the string only indirectly.

Code: Select all

set LF=^


rem the two empty lines are important (without any spaces)
for /F %%a in ('copy /Z "%~f0" nul') DO set "cr=%%a"
set "map=!map:~0,10!!LF!!map:~11!"
set "map=!map:~0,13!!CR!!map:~14!"

hope it helps
jeb

[dbenham]

Thanks again Jeb.

What about the other non NULL "problematic" characters? I'm still stumped on how to introduce them.

I have some concern with 0x0A (Ctrl-Z). Is this no longer treated as an end of file marker? I haven't done any testing, I'm just remembering issues I ran into back in the 80's.

One other thing: The :asc linear search through the map seems inefficient. I thought I could employ a search mechanism similar to what is used for the :strLen function. But I hit a snag in that the collation sequence of the characters does not match the ASCII code sequence. Is there a way to temporarily change the collation rules of string comparisons from within a batch file?

[dbenham]

...and for the TAB character, I can get that character in my source, but my programming editor is typically set to convert tabs into spaces. Eventually I'm sure I would make a mistake and forget to disable this feature when editing the source, and my map would become corrupted.

Any way to programmatically generate a tab character without it appearing in the source?

[dbenham]

...and I'm back to thinking many of the "problematic" characters are still a problem.

The problematic characters can be handeld with delayed expansion.

I can't use delayed expansion in my endlocal block that returns the results. This was what I meant in my original post when I said "I'm not sure [putting the problem characters in the map variable] would be productive for this application."

[jeb]

...and I'm back to thinking many of the "problematic" characters are still a problem.

The "problematic" are not so problematic ...

I can't use delayed expansion in my endlocal block that returns the results. This was what I meant in my original post when I said "I'm not sure [putting the problem characters in the map variable] would be productive for this application."

You can simply use the delayed expansion to return a var from a local block with the FOR-Endlocal technic.

Code: Select all

:CreateLF varByRef
setlocal
set LF=^


for %%a in ("!lf!") do (
   endlocal
   set "%~1=%%~a"
)
goto :eof

jeb

[dbenham]

Simple solution, and powerfull.
Thanks Jeb!

What about the other issues in my prior back-to-back posts?

I still can't get the other chars in my source code (editor limitation). But even if I do I'm worried what will happen when the command line interprets the text at execution time. For example, will <BS> not simply erase the prior character in the variable?

conern over <Ctrl-Z> as end of file marker - is this valid?

If I could get the chars in my source I would test, but I'm stuck. Any suggestions here?

Finally, any thoughts on how to change character collation so I can improve :asc asciiMap lookup performance?

Thanks for your help
Dave

[jeb]

Ok, I will try ...

first the characters

0x08 = DEL - can be produced with the prompt method
0x09 = TAB - didn't know how to produce it (never tried it, my editor didn't remove it)
0x0A = LF - can be produced with the multiline method
0x0D = CR - can be produced with the buffer overlflow or the copy /z method
0x1A = EOF/CTRL-Z - can be produced with the copy /a nul+nul method
0x1B = ESC - can be produced with the prompt method

Code: Select all

:BL.String.CreateLF
:: Creates a variable with one character LF=Ascii-10
:: LF should  be used later only with DelayedExpansion
set LF=^


rem ** The two empty lines are neccessary, spaces are not allowed
rem ** Creates a percent variant "NLM=^LF", but normaly you should use the !LF! variant
set ^"NLF=^^^%lf%%lf%^%lf%%lf%^"
goto :eof

:BL.String.CreateDEL_ESC
:: Creates two variables with one character DEL=Ascii-08 and ESC=Ascii-27
:: DEL and ESC can be used  with and without DelayedExpansion
setlocal
for /F "tokens=1,2 delims=#" %%a in ('"prompt #$H#$E# & echo on & for %%b in (1) do rem"') do (
  ENDLOCAL
  set "DEL=%%a"
  set "ESC=%%b"
  goto :EOF
)
goto :eof

:BL.String.CreateCR
::: CR should  be used only with DelayedExpansion
for /F "usebackq" %%a in (`copy /Z "%~dpf0" nul`) DO (
   set "cr=%%a"
)
goto :eof

:BL.String.CreateSUB
::: SUB = 0x1A can be used with or without DelayedExpansion
copy /A nul+nul sub.tmp > nul
for /F %%a in (sub.tmp) DO (
   set "char.sub=%%a"
)
del sub.tmp
goto :eof

The DEL(0x08) didn't remove the content of a variable, but you can use it to remove characters on the screen.

Code: Select all

<nul set /p ".=123456"
echo !DEL!!DEL!AB
---- OUTPUT ---
1234AB

The DEL(0x08) didn't remove the content of a variable, but you can use it to remove characters on the screen.
Batch handle it like any other character

Code: Select all

<nul set /p ".=123456"
echo !DEL!!DEL!AB
---- OUTPUT ---
1234AB

CR/LF can be produced, can be handeld as variable content but causes many problems/new behaviour.

Code: Select all

<nul set /p ".=123456"
echo !CR!AB
echo one!LF!two
---- OUTPUT ---
AB3456
one
two

EOF/CTRL-Z=0x1A can't be set in a variable directly, but with the FOR /F read from a file it works.

All characters except of NUL can be read from a file, therefore I would move the map into an external file.
And read it with a FOR /F into a variable.

To build the characters I use debug.exe or xvi32 (a hex-editor), or a small jscript.

To be faster, I would only once build the map, and reuse it everytime.

jeb

[dbenham]

Thanks Jeb

I'll try out your suggestions in a few days. I'm heading out of town in a few minutes and won't have access to a computer.

Dave

[plp626]

0x09 = TAB - didn't know how to produce it (never tried it, my editor didn't remove it)

=========================
TAB can be produced with the 'cacls|find " /R user"' or 'reg query hkcu\environment /v temp |find /i "temp"' method

here is the function （my OS is XP. test OK!）

Code: Select all

:BL.String.CreateTAB
For /F "skip=4 delims=pR tokens=1,2" %%a In (
    'reg query hkcu\environment /v temp'
)Do if "%~1"=="" (set TAB=%%b)else set "%~1=%%b"
goto :eof

[jeb]

Very nice,
I tested also some commands, but can't find any TAB-producer.

I'm tested your solutions with XP and Vista, the "reg query" method fails under Vista.

The calcs way is also ok, but could be a bit more complicated, as the "find User" is very localized (and doesn't work on my system).

Perhaps it's possible to capture the lines and find the <TAB> without using a static search string.

always amused about the complexity of batch
jeb

[aGerman]

Both don't work on Win7. It seems M$ replaced all TABs by spaces. I didn't found any command which output a TAB character yet.

Regards
aGerman

[aGerman]

[EDIT]
Found a command on Win7: shutdown /? displays the following on my PC
..... snip

Code: Select all

 U     0   0   Anderer Grund (nicht geplant)
E      0   0   Anderer Grund (nicht geplant)
E P    0   0   Anderer Grund (geplant)
 U     0   5   Anderer Fehler: System reagierte nicht mehr
E      1   1   Hardware: Wartung (nicht geplant)
E P    1   1   Hardware: Wartung (geplant)
E      1   2   Hardware: Installation (nicht geplant)
E P    1   2   Hardware: Installation (geplant)
E      2   2   Betriebssystem: Wiederherstellung (geplant)
E P    2   2   Betriebssystem: Wiederherstellung (geplant)
  P    2   3   Betriebssystem: Aktualisierung (geplant)
E      2   4   Betriebssystem: Neukonfigurierung (nicht geplant)
E P    2   4   Betriebssystem: Neukonfigurierung (geplant)
  P    2   16   Betriebssystem: Service Pack (geplant)
       2   17   Betriebssystem: Hotfix (nicht geplant)
  P    2   17   Betriebssystem: Hotfix (geplant)
       2   18   Betriebssystem: Sicherheitspatch (nicht geplant)
  P    2   18   Betriebssystem: Sicherheitspatch (geplant)
E      4   1   Anwendung: Wartung (nicht geplant)
E P    4   1   Anwendung: Wartung (geplant)
E P    4   2   Anwendung: Installiert (geplant)
E      4   5   Anwendung: Reagiert nicht
E      4   6   Anwendung: Instabil
 U     5   15   Systemfehler: Abbruchfehler
 U     5   19   Sicherheitsproblem
E      5   19   Sicherheitsproblem
E P    5   19   Sicherheitsproblem
E      5   20   Netzwerkkonnektivität getrennt (nicht geplant)
 U     6   11   Stromversorgungsfehler: Kabel entfernt
 U     6   12   Stromversorgungsfehler: Umgebung
  P    7   0   Herunterfahren von Legacy-API

..... snip

I found a TAB character before, between and behind the numbers.

This code works for me:

Code: Select all

@echo off &setlocal

for /f "tokens=2 delims=1234567890" %%a in ('shutdown /?^|findstr /bc:"E"') do set "TAB=%%a"

echo text%TAB%text
pause

... but should be tested on other PC's and other Windows versions.

Regards
aGerman

[/EDIT]

[jeb]

Yes, you found it.

I tested it with XP and Vista and both works (Both also in a german version).

And I build a code to get the TAB without findstr for a constant text or number.
It uses the FOR /F delims functionality to split at spaces and TAB.
First I split, then I replace the spaces and then I split again, a delim can be only a TAB then.

Code: Select all

@echo off
setlocal 
call :BL.Strings.CreateTAB
echo TAB=#%tab%#
goto :eof

:BL.Strings.CreateTAB
setlocal EnableDelayedExpansion
set "TAB=#"
for /F "tokens=*" %%a in ('shutdown /?') do (
   set "line=%%a"
   set "line=!line: =!"
   for /F "tokens=1,2*" %%x in ("!line!") do (
     set "split=!line:%%x=!"
      if "%%y" NEQ "" (      
         set "tab=!split:~0,1!"
         goto :tab_found
      )
   )
)
:tab_found
(
   endlocal
   set "TAB=%tab%"
   goto :eof
)

goto :eof

jeb

[aGerman]

Good idea, jeb!
Honestly I'm not sure whether the E is language independent or not. Maybe I can check this at work. The half of the commands are in English, hopefully shutdown is one of them.

Regards
aGerman