Page 1 of 1

base128/255 help on this code

Posted: 04 Oct 2013 14:25
by einstein1969
Hi,

I have a problem with this code in the management of "poison" character!

I not skilled for this!

Help needed , thanks.

This code get a string and convert in another base. For now 128 but the goal is higher than 128 (near the maximum which is 255 i think)

In this manner is possible (i thinks) rappresent all 256 char (or integer) in the dos batch.

If there 'a better way I'd like to know!

The initial problem is in the decode128 subroutine in the part that get the position of the character in the new "ascii" system

Code: Select all

@echo off & setlocal EnableDelayedExpansion & goto :Init_System

:Main

set "buffer=123456789"

call :encode128 buffer encoded

echo !encoded!

call :decode128 encoded decoded

echo !decoded!


goto :EndMain

:encode128 buffer
  setlocal EnableDelayedExpansion

   set buffer=!%~1!& set "ascii=!#$%()*,.0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"

   call :strlen Buffer size
   echo "!buffer!" size:!size!

   set /a size2=size+1, ls=0, rs=7, r=0
   set "encoded="

   set inx=0

:encode128_loop1

   echo ls:!ls! inx:!inx!

   if !ls! gtr 7 (
      set /a inx=inx-1, ls=0, rs=7
   )

   rem for %%c in (buffer:~!inx!,1) set c=!%%c!
   
   echo "!buffer!" inx:!inx!
   if !inx! lss !size! (call charlib asc buffer !inx! nc) else set nc=0

   set r1=nc
   set /a    "nc=(nc << ls), nc=(nc & 0x7F)|r, r=(r1>>rs) & 0x7F, ls=ls+1, rs-=1"

   for %%c in ("ascii:~!nc!,1") do set encoded=!encoded!!%%~c!

   
   set /a inx=inx+1

   if !inx! lss !size2! goto :encode128_loop1

( endlocal
   set %2=%encoded%
)
goto :eof

:decode128
  setlocal EnableDelayedExpansion
            rem            1         2         3         4         5         6         7         8         9         0         1         2
            rem  01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567
   set buffer=!%~1!& set "ascii=!#$%()*,.0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"

   call :strlen Buffer size
   echo "!buffer!" size:!size!

   set /a ls=7, rs=8, r=0
   
   set "decoded="

   set inx=0

:decode128_loop1

   echo ls:!ls! inx:!inx!

   for %%c in ("buffer:~!inx!,1") do set c=!%%~c!

   set "nc="

   if "!c!"==";" set nc=20
       if "!c!"=="=" set nc=21
       if "!c!"=="^!" set nc=0

   if not defined nc for /l %%i in (0,1,127) do (
         if "!ascii:~%%i,1!"=="!c!" set /a nc=%%i
         echo if "!ascii:~%%i,1!"=="!c!" set /a nc=%%i
         pause
      )
   if not defined nc echo("!c!" not found in "!ascii!"

   if !rs! gtr 7 (
           set rs=1, ls=7, r=nc
         ) else (
      set /a "r1=nc, nc=(nc<<ls) & 0xFF, nc=nc|r, r=r1>>rs, rs=rs+1, ls=ls-1;"

      call charlib chr nc c
      set decoded=!decoded!!c!
    )

   if !inx! lss !size! goto :decode128_loop1

( endlocal
   set %2=%decoded%
)
goto :eof


Qui ci vanno le subroutine/fuctions


:strLen string len -- returns the length of a string
::                 -- string [in]  - variable name containing the string being measured for length
::                 -- len    [out] - variable to be used to return the string length
:: Many thanks to 'sowgtsoi', but also 'jeb' and 'amel27' dostips forum users helped making this short and efficient
:$created 20081122 :$changed 20101116 :$categories StringOperation
:$source http://www.dostips.com
(   SETLOCAL ENABLEDELAYEDEXPANSION
    set "str=A!%~1!"&rem keep the A up front to ensure we get the length and not the upper bound
                     rem it also avoids trouble in case of empty string
    set "len=0"
    for /L %%A in (12,-1,0) do (
        set /a "len|=1<<%%A"
        for %%B in (!len!) do if "!str:~%%B,1!"=="" set /a "len&=~1<<%%A"
    )
)
( ENDLOCAL & REM RETURN VALUES
    IF "%~2" NEQ "" SET /a %~2=%len%
)
EXIT /b


::_____________________________________________________________
:difftime t1var t2var result
::
:: Usage:
::    set t1=%time%
::       rem do some stuff
::    set t2=%time%
::    Call :difftime t1 t2

  setlocal EnabledelayedExpansion

   set "t=!%~1: =0! !%~2: =0!"

   set /a c=(((1!t:~12,2!-1!t:~0,2!)*60+(1!t:~15,2!-1!t:~3,2!))*60+(1!t:~18,2!-1!t:~6,2!))*100+(1!t:~-2!-1!t:~9,2!)

   if !c! lss 0 set /a c+=24*60*60*100

   if "%~3" equ "" set "p=0!c!"

( endlocal & rem return value
  if "%~3" neq "" (set "%~3=%c%") else ( echo %t% Elapsed: %c:~0,-2%.%p:~-2% seconds)
)
goto :eof

::_____________________________________________________________
:difftime2 t1var t2var
::
:: leggermente più lenta

  setlocal EnabledelayedExpansion
   
   for /F "tokens=1-8 delims=:.," %%a in ("!%~1: =0!,!%~2: =0!") do set /a "d=((((1%%e-1%%a)*60)+1%%f-1%%b)*60+1%%g-1%%c)*100+1%%h-1%%d

   if !d! lss 0 set /a d+=24*60*60*100

   set "tp=0!d!" & echo !%~1! !%~2! Elapsed: !d:~0,-2!.!tp:~-2! seconds

  endlocal
goto :eof

:ReserveEnvironmentSpace sizeInKB
   rem Define the first large variable (reserving 6 bytes for variable name)
   set z1=X
   for /L %%i in (1,1,12) do set z1=!z1!!z1!
   set z1=!z1!!z1:~8!
   rem Define the rest of large variables
   set /A lastVar=%1 / 8
   for /L %%i in (2,1,%lastVar%) do set z%%i=!z1!
   rem Delete all the large variables in bottom-up order
   for /L %%i in (%lastVar%,-1,1) do set z%%i=
Goto :eof


::_____________________________________________________________
:Init_System

:: Why does SET performance degrade as environment size gro
:: http://www.dostips.com/forum/viewtopic.php?f=3&t=2597&start=15

call :ReserveEnvironmentSpace 1290

:: rem per aggiungere caratteri oltre il 126 usare: alt+0128 e non alt+128

Goto :Main

::_____________________________________________________________
:EndMain

Goto :eof


Einstein1969

Re: base128/255 help on this code

Posted: 05 Oct 2013 03:51
by penpen
I assume your only problem is this (i haven't searched for more, if there are additional issues, just mark them):

Code: Select all

set "ascii=!#$%()*,.0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"

The three problematic characters here are '!', '%', and '^'.
First the your code expands the %variable%, the 'missing' '%' is assumed/appended at the end.
As the variable is undefined it prints the part after the ':' character.
Then the !delayed expanded variable! is expanded, this time the missing '!' is not appended and the text after remain unchanged, but the '!' character disappears.
After that it expands the ^X to X, where X is any character.

Code: Select all

:: code
set "ascii=!#$%()*,.0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"
:: step 1
set "ascii=!#$;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"
:: step 2
set "ascii=#$;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"
:: step 3
set "ascii=#$;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"

So if you want the assign the string unchanged to ascii you have to avoid all three steps, that means:
The '%' is the variable expansion character, just replace it with a valid expression, thats result is a single '%' (%%).
And to avoid the delayedExpansion got to work on the first ! you have to escape it with the escape character (^!).
The '^' has to be escaped, too, so just add a second '^'.
So this is what you seem to need:

Code: Select all

set "ascii=^!#$%%()*,.0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"

penpen

Edit: Steps are in the wrong order, i fixed it.
Edit2: Adjusted the text to meet the steps.

Re: base128/255 help on this code

Posted: 05 Oct 2013 04:26
by aGerman
dbenham already provided a code using this method. Just search the forum.

Another method that I remember was using FORFILES. It has some issues but most of the characters are displayed correctly.

Code: Select all

@echo off &setlocal

set "hex=0 1 2 3 4 5 6 7 8 9 A B C D E F"
for %%i in (%hex%) do (
  for %%j in (%hex%) do (
    call :byte2char "0x%%i%%j"
  )
)

pause
exit /b

:byte2char hexstr
set "c="
for /f delims^=^ eol^= %%i in (
  'forfiles /p "%~dp0." /m "%~nx0" /c "cmd /c echo(^%~1"'
) do set "c=%%i"
setlocal EnableDelayedExpansion
echo %1 "!c!"
endlocal
exit /b

Regards
aGerman

Re: base128/255 help on this code

Posted: 05 Oct 2013 07:52
by einstein1969
@penpen
thanks a lot, i have correct and now seem ok. Explained very well 8)

@german
Thanks , i searched but there are too many thread :D.
you're thinking about rot13?

Thanks for the alternative, in case there are no solutions, or if the solutions found to be less efficient it will be adopted. Very Good!!!

@all

The problem i now in the encode128. I have tried to return a variable with "!" but with no success :( :twisted:

I have added a enc2 with a caret... but the caret "^" don't appear. :roll:

Help needed.

Einstein1969

Re: base128/255 help on this code

Posted: 05 Oct 2013 08:26
by aGerman
I had this thread in mind:
new functions: :chr, :asc, :asciiMap
You'll find some links at the end of the thread since the forum software doesn't display some of the characters correctly.

Regards
aGerman

Re: base128/255 help on this code

Posted: 05 Oct 2013 09:46
by penpen
einstein1969 wrote:The problem i now in the encode128. I have tried to return a variable with "!" but with no success :( :twisted:
Hmm, this should be no problem if you use carets:

Code: Select all

@echo off
setlocal
call :test
echo %test%
endlocal
goto:eof

:test
   setlocal enableDelayedExpansion
   set "test=^!abcde^!"
   endlocal & set "test=%test%"
   goto:eof
Maybe another character is problematic.

penpen

Re: base128/255 help on this code

Posted: 05 Oct 2013 09:58
by einstein1969
penpen wrote:
einstein1969 wrote:The problem i now in the encode128. I have tried to return a variable with "!" but with no success :( :twisted:
Hmm, this should be no problem if you use carets:

Code: Select all

@echo off
setlocal
call :test
echo %test%
endlocal
goto:eof

:test
   setlocal enableDelayedExpansion
   set "test=^!abcde^!"
   endlocal & set "test=%test%"
   goto:eof
Maybe another character is problematic.

penpen


I have run the above

tmp.cmd:

Code: Select all

@echo off
setlocal
call :test
echo %test%
endlocal
goto:eof

:test
   setlocal enableDelayedExpansion
   set "test=^!abcde^!"
   endlocal & set "test=%test%"
   goto:eof


result:

Code: Select all

E:\x264\provini>tmp.cmd
ECHO disattivato.

E:\x264\provini>


:?:

on windows 7 32bit


Einstein1969

Re: base128/255 help on this code

Posted: 05 Oct 2013 10:41
by penpen
Sorry, my code works on Win XP 32 bit SP3... .
It is a little bit strange, that this works different using Win 7 32 bit... .
Maybe the initial state differs, when using cmd files:
Rename your file to test.bat and try this on dos shell

Code: Select all

cmd /E:ON  /V:ON  /C test
cmd /E:ON  /V:OFF /C test
cmd /E:OFF /V:ON  /C test
cmd /E:OFF /V:OFF /C test
Please post the results. The second case should work.
If it does you may change the test code to:

Code: Select all

@echo off
setlocal disableDelayedExpansion
call :test
echo %test%
endlocal
goto:eof

:test
   setlocal enableDelayedExpansion
   set "test=^!abcde^!"
   endlocal & set "test=%test%"
   goto:eof
and try if it then works as a cmd file.

Or you may test the changed code directly, but i'm a little bit curious. :wink:

penpen

Edit: If none of the cases work, i fear i can't help, as i have no win7, and it is unclear to me why it doesn't work.
Edit2: Changed the testing code to have less work.

Re: base128/255 help on this code

Posted: 05 Oct 2013 11:06
by einstein1969
ok! was the initial state. Thanks.

result:

Code: Select all

E:\x264\provini>type tmp.cmd
@echo off
setlocal
call :test
echo %test%
endlocal
goto:eof

:test
   setlocal enableDelayedExpansion
   set "test=^!abcde^!"
   endlocal & set "test=%test%"
   goto:eof

E:\x264\provini>ren tmp.cmd test.cmd

E:\x264\provini>cmd /E:ON  /V:ON
Microsoft Windows [Versione 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. Tutti i diritti riservati.

E:\x264\provini>test
ECHO disattivato.

E:\x264\provini>cmd /E:ON  /V:OFF
Microsoft Windows [Versione 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. Tutti i diritti riservati.

E:\x264\provini>test
!abcde!

E:\x264\provini>cmd /E:OFF /V:ON
Microsoft Windows [Versione 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. Tutti i diritti riservati.

E:\x264\provini>test
ECHO disattivato.
Impossibile trovare l'etichetta batch specificata - eof

E:\x264\provini>cmd /E:OFF /V:OFF
Microsoft Windows [Versione 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. Tutti i diritti riservati.

E:\x264\provini>test
ECHO disattivato.
Impossibile trovare l'etichetta batch specificata - eof

E:\x264\provini>notepad test.cmd

E:\x264\provini>type test.cmd
@echo off
setlocal disableDelayedExpansion
call :test
echo %test%
endlocal
goto:eof

:test
   setlocal enableDelayedExpansion
   set "test=^!abcde^!"
   endlocal & set "test=%test%"
   goto:eof

E:\x264\provini>test.cmd
ECHO disattivato.
Impossibile trovare l'etichetta batch specificata - eof

E:\x264\provini>exit

E:\x264\provini>test.cmd
ECHO disattivato.
Impossibile trovare l'etichetta batch specificata - eof

E:\x264\provini>exit

E:\x264\provini>test.cmd
!abcde!

E:\x264\provini>exit

E:\x264\provini>test.cmd
!abcde!

E:\x264\provini>


I thinks I forgot last exit but i remember was like the last above.

why i need to start with delayed variable disabled?

Einstein1969

Re: base128/255 help on this code

Posted: 05 Oct 2013 12:22
by penpen
You need to start with delayed expansion off, because of the set instruction in the "endlocal & set..." line:

Code: Select all

:test
   setlocal enableDelayedExpansion
   set "test=^!abcde^!"
   endlocal & set "test=%test%"
   goto:eof
The "setlocal enableDelayedExpansion" will enable it, but the endlocal recovers the previous state.
If the delayed Expansion is enabled after the above endlocal, the set "test=..:" will compute this way:

Code: Select all

:: assumed test is !abcde!
test=%test%
:: step 1 %variable% expansion
test=!abcde!
:: step 2 !delayed_variable! expansion, assumed variable abcde is not defined.
test=

If delayed expansion is disabled after the endlocal then the step 2 is not executed, so it contains the test value.

penpen

Re: base128/255 help on this code

Posted: 05 Oct 2013 15:10
by einstein1969
@penpen
ok , now is clear. For now the example work.

this part changed:

Code: Select all

@echo off & setlocal EnableDelayedExpansion & goto :Init_System

Abstract layer
Code/decode in different base! (For compress environment/var , binary working, array of float/integer/strings , etc.)

For now base 128, goal is:

1) near base 255
2) complete run in memory (environment)
3) Fast execute
4) simple use...

:Main

setlocal disabledelayedexpansion

   set "buffer=123456789"

   call :encode128 buffer encoded

   setlocal Enabledelayedexpansion

      echo "!buffer!" -^> "!encoded!"

   endlocal

   call :decode128 encoded decoded

   setlocal Enabledelayedexpansion

      echo "!encoded!" -^> "!decoded!"

      if "!buffer!"=="!decoded!" (echo ok.) else echo No ok.

   endlocal

endlocal

goto :EndMain

:encode128 buffer
  setlocal EnableDelayedExpansion

   set buffer=!%~1!& set "base128=^!#$%%()*,.0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"

   call :strlen Buffer size & rem this is not necessary?!

   set /a ls=0, rs=7, r=0, inx=0

   set "encoded="

:encode128_loop1

   if !ls! gtr 7 set /a inx=inx-1, ls=0, rs=7
   
   if !inx! lss !size! (call charlib asc buffer !inx! nc) else set nc=0

   set /a    "r1=nc, nc=(nc << ls), nc=(nc & 0x7F)|r, r=(r1>>rs) & 0x7F, ls+=1, rs-=1"
   
   for %%c in ("base128:~!nc!,1") do set "c=!%%~c!"

   set "encoded=!encoded!!c!"

   set /a inx=inx+1

   if !inx! leq !size! goto :encode128_loop1

( endlocal
   set "%~2=%encoded%"
)
goto :eof

:decode128
  setlocal EnableDelayedExpansion
   rem                         1         2         3         4         5         6         7         8         9         0         1         2
   rem                01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567
   rem set buffer=!%~1!& set "base128=!#$%()*,.0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"
   rem escape of ! % ^
   set buffer=!%~1!& set "base128=^!#$%%()*,.0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^^_abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ"

   call :strlen Buffer size & rem this is not necessary?!

   set /a ls=7, rs=8, r=0, inx=0
   
   set "decoded="

:decode128_loop1

   for %%c in ("buffer:~!inx!,1") do set c=!%%~c!

   set "nc="

   rem this part can be optimized by sort of char
   if not defined nc for /l %%i in (0,1,127) do if "!base128:~%%i,1!"=="!c!" set /a nc=%%i

   if not defined nc echo("!c!" not found in "!base128!" Error in implementation.

   if !rs! gtr 7 (set /a rs=1, ls=7, r=nc, inx+=1
   ) else (
      set /a "r1=nc, nc=(nc<<ls) & 0xFF, nc=nc|r, r=r1>>rs, rs+=1, ls-=1, inx+=1"
      call charlib chr nc c
      set decoded=!decoded!!c!
   )

   if !inx! lss !size! goto :decode128_loop1

( endlocal
   set %2=%decoded%
)
goto :eof


but I think I will replace the exclamation point.

For now it 's possible to represent any information potentially losing about 13% of space. But still better than the hexadecimal or decimal format. Can also be useful for your project on the float (that great you are!)

@aGerman
Thanks for the link. I'm reading... The function byte2char seems to have the same limitation of 0x00? and could possibly be used in place of charlib to create the map ascii. But then always need a mechanism to map the 0x00, or am I wrong?

Einstein1969

Re: base128/255 help on this code

Posted: 05 Oct 2013 16:07
by aGerman
There is no way to assign, read or output the 0x00 character in Batch. The reason why is that cmd.exe uses this character as string terminator in the computers memory.

Regards
aGerman