Need help copying a line of text from a file to another (without mangling it) :AppendFileLineToFile

[shodan]

Hello

I made this function to copy arbitrary lines of text, from one file to another

Code: Select all

::Usage Call :AppendFileLineToFile inputfile outputfile 3 4 50-75 5 6 7 ... N
:AppendFileLineToFile
set "_AppendFileLineToFile_prefix=_AFLTF"
set "_AFLTF_InputFile=%~1"
set "_AFLTF_OutputFile=%~2"
:AppendFileLineToFile-arg
for /f "delims=- tokens=1,2" %%a in ("%~3") do ( set "_AFLTF_Start=%%a" & set "_AFLTF_Stop=%%b"  )
if not defined _AFLTF_Stop set /a _AFLTF_Stop=%_AFLTF_Start%
Setlocal enabledelayedexpansion
if %_AFLTF_Start% GTR 1 set /a "_AFLTF_skip=%_AFLTF_Start%-1"
if %_AFLTF_Start% GTR 1 ( set "_AFLTF_skip=skip^=%_AFLTF_skip%^" ) else ( set "_AFLTF_skip=" )
for /f %_AFLTF_skip% delims^=^ eol^= %%a in (' ^( type "%_AFLTF_InputFile%" ^| %SystemRoot%\System32\findstr /N /R /C:".*" ^) 2^>nul ') do ( 
	for /f "delims=:" %%f in ("%%a") do if %%f GTR %_AFLTF_Stop% GoTo :AppendFileLineToFile-end
	set _AFLTF_buffer=%%a
	if defined _AFLTF_buffer >>"%_AFLTF_OutputFile%" echo(!_AFLTF_buffer:*:=!
	) 
endlocal
:AppendFileLineToFile-end
if "[%~4]" NEQ "[]" ( shift & GoTo :AppendFileLineToFile-arg )
Call :ClearVariablesByPrefix %_AppendFileLineToFile_prefix% _AppendFileLineToFile_prefix & GoTo :EOF

The logic of copying the correct rows to the right place LARGELY works
Unfortunately, it does damage some lines of text and it completely breaks if it encounters ">>" possibly other characters


For example this line

Code: Select all

for /f delims^=:^ tokens^=1 %%a in ('%SystemRoot%\System32\findstr /I /N "^:%~2" "%~1" ^| findstr /I /V "::%~2[a-zA-Z0-9\-\/\?\!\@\%%\$\#\^\*\)\{\}\[\]\:\_]"') do ( if "[%~3]" NEQ "[]" set "%~3=%%a" & exit /b %%a )

becomes this line

Code: Select all

for /f delims=: tokens=1 %%a in ('%SystemRoot%\System32\findstr /I /N ":%~2" "%~1" | findstr /I /V "::%~2[a-zA-Z0-9\-\/\?\\_]"') do ( if "[%~3]" NEQ "[]" set "%~3=%%a" & exit /b %%a )

And this line

Code: Select all

	if defined _AFLTF_buffer >>"%_AFLTF_OutputFile%" echo(!_AFLTF_buffer:*:=!

breaks it in a very strange way that I do not understand.

Here is a screenshot
The original file is on the right, the newly created on the left

notepad++_dx5YxoUAgP.png (615.34 KiB) Viewed 13762 times

This is part of a larger project, which allows copying selected batch functions to another batch file.

I have created a special "function switching" batch file, when you call it, the first argument is the function that will be executed with the rest of your arguments.

Here I demonstrate the "listfunctions" function, which outputs a list of all function existing in a batchfile

Code: Select all

bfw listfunctions bfw.bat

ShiftedArgumentCaller AddFunctionToBatch GetLastToken IsLastToken GetLabelsOnly IsFile GetFunctionRows GetLabelRow GetFunctionExit GetFunctionPreambleRow GetFunctionPostscriptRow ClearVariablesByPrefix GetFunctionName GetBatchCore GetNextExitRow ListFunctions GetNextFunctionName GetNextFunctionRow GetPreviousExitRow GetEOFrow countLines IsFunctionLabelExcluded AppendFileLineToFile AppendFileLineToFile-arg GetPreviousEmptyRow GetNextEmptyRow BFWFunctionSwitcher-text EndOF_BFWFunctionSwitcher-text

And now I demontrate the "AddFunctionToBatch" function, which copies chosen functions from a batchfile to another batchfile. It ALMOST works

Code: Select all

bfw AddFunctionToBatch test\mybfw.bat CORE ShiftedArgumentCaller AddFunctionToBatch GetLastToken IsLastToken GetLabelsOnly IsFile GetFunctionRows GetLabelRow GetFunctionExit GetFunctionPreambleRow GetFunctionPostscriptRow ClearVariablesByPrefix GetFunctionName GetBatchCore GetNextExitRow ListFunctions GetNextFunctionName GetPreviousExitRow GetEOFrow countLines IsFunctionLabelExcluded AppendFileLineToFile AppendFileLineToFile-arg GetPreviousEmptyRow GetNextEmptyRow BFWFunctionSwitcher-text

This previous command, creates the test\mybfw.bat file, which contains (almost) everything it needs to perform the task of the original bfw.bat file.

Unfortunately the functions :GetLabelRow and :AppendFileLineToFile are mangled by this process

The original bfw.bat

https://pastebin.com/456kfhTi

The new test\mybfw.bat file

https://pastebin.com/vkkMGin9

You can also observe the bfw.bat file at

https://github.com/batchfileframework/B ... fw/bfw.bat

I also include it as a zip here

bfw.zip

(8.57 KiB) Downloaded 728 times

[shodan]

Looking in my older code, I found a function called "SimpleFileToArray" that takes lines of text from a file and loads them in an array of variable

I just tested it and I think it might work for this.

Here is the function itself.

Code: Select all

::Usage Call :SimpleFileToArray OutputArray Filename
:SimpleFileToArray
set /a "%~1.lbound=%%f"
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~2"') do ( 
	for /f "tokens=1,2* delims=:" %%f in ("%%a") do set /a "%~1.ubound=%%f" & set %~1[%%f]=%%a
	)
set /a "_SFTA_index=1"
call set /a "_SFTA_ubound=%%%~1.ubound%%"
:SimpleFileToArray-loop
setlocal enabledelayedexpansion
set _SFTA_localscope=true
set %~1[%_SFTA_index%]=!%~1[%_SFTA_index%]:*:=!
for /f "delims=" %%a in ('set %~1[%_SFTA_index%] 2^>nul') do (
		endlocal
		set %%a
	)
if defined _SFTA_localscope endlocal & set %~1[%_SFTA_index%]=
set /a "_SFTA_index+=1"
if %_SFTA_index% LEQ %_SFTA_ubound% GoTo :SimpleFileToArray-loop
GoTo :EOF

And here is what happens if I read the previous file

The problematic line 150 of :GetLabelRow, was as follows

Code: Select all

for /f delims^=:^ tokens^=1 %%a in ('%SystemRoot%\System32\findstr /I /N "^:%~2" "%~1" ^| findstr /I /V "::%~2[a-zA-Z0-9\-\/\?\!\@\%%\$\#\^\*\)\{\}\[\]\:\_]"') do ( if "[%~3]" NEQ "[]" set "%~3=%%a" & exit /b %%a )

It became

Code: Select all

bfw[150]=for /f delims^=:^ tokens^=1 %%a in ('%SystemRoot%\System32\findstr /I /N "^:%~2" "%~1" ^| findstr /I /V "::%~2[a-zA-Z0-9\-\/\?\!\@\%%\$\#\^\*\)\{\}\[\]\:\_]"') do ( if "[%~3]" NEQ "[]" set "%~3=%%a" & exit /b %%a )

Great, it's a match !


And looking at the larger function


It was :AppendFileLineToFile

Code: Select all

::Usage Call :AppendFileLineToFile inputfile outputfile 3 4 50-75 5 6 7 ... N
:AppendFileLineToFile
set "_AppendFileLineToFile_prefix=_AFLTF"
set "_AFLTF_InputFile=%~1"
set "_AFLTF_OutputFile=%~2"
:AppendFileLineToFile-arg
for /f "delims=- tokens=1,2" %%a in ("%~3") do ( set "_AFLTF_Start=%%a" & set "_AFLTF_Stop=%%b"  )
if not defined _AFLTF_Stop set /a _AFLTF_Stop=%_AFLTF_Start%
Setlocal enabledelayedexpansion
if %_AFLTF_Start% GTR 1 set /a "_AFLTF_skip=%_AFLTF_Start%-1"
if %_AFLTF_Start% GTR 1 ( set "_AFLTF_skip=skip^=%_AFLTF_skip%^" ) else ( set "_AFLTF_skip=" )
for /f %_AFLTF_skip% delims^=^ eol^= %%a in (' ^( type "%_AFLTF_InputFile%" ^| %SystemRoot%\System32\findstr /N /R /C:".*" ^) 2^>nul ') do ( 
	for /f "delims=:" %%f in ("%%a") do if %%f GTR %_AFLTF_Stop% GoTo :AppendFileLineToFile-end
	set _AFLTF_buffer=%%a
	if defined _AFLTF_buffer >>"%_AFLTF_OutputFile%" echo(!_AFLTF_buffer:*:=!
	) 
endlocal
:AppendFileLineToFile-end
if "[%~4]" NEQ "[]" ( shift & GoTo :AppendFileLineToFile-arg )
Call :ClearVariablesByPrefix %_AppendFileLineToFile_prefix% _AppendFileLineToFile_prefix & GoTo :EOF

And in the array it is

Code: Select all

bfw[283]=::Usage Call :AppendFileLineToFile inputfile outputfile 3 4 50-75 5 6 7 ... N
bfw[284]=:AppendFileLineToFile
bfw[285]=set "_AppendFileLineToFile_prefix=_AFLTF"
bfw[286]=set "_AFLTF_InputFile=%~1"
bfw[287]=set "_AFLTF_OutputFile=%~2"
bfw[288]=:AppendFileLineToFile-arg
bfw[289]=for /f "delims=- tokens=1,2" %%a in ("%~3") do ( set "_AFLTF_Start=%%a" & set "_AFLTF_Stop=%%b"  )
bfw[290]=if not defined _AFLTF_Stop set /a _AFLTF_Stop=%_AFLTF_Start%
bfw[291]=Setlocal enabledelayedexpansion
bfw[292]=if %_AFLTF_Start% GTR 1 set /a "_AFLTF_skip=%_AFLTF_Start%-1"
bfw[293]=if %_AFLTF_Start% GTR 1 ( set "_AFLTF_skip=skip^=%_AFLTF_skip%^" ) else ( set "_AFLTF_skip=" )
bfw[294]=for /f %_AFLTF_skip% delims^=^ eol^= %%a in (' ^( type "%_AFLTF_InputFile%" ^| %SystemRoot%\System32\findstr /N /R /C:".*" ^) 2^>nul ') do (
bfw[295]=       for /f "delims=:" %%f in ("%%a") do if %%f GTR %_AFLTF_Stop% GoTo :AppendFileLineToFile-end
bfw[296]=       set _AFLTF_buffer=%%a
bfw[297]=       if defined _AFLTF_buffer >>"%_AFLTF_OutputFile%" echo(!_AFLTF_buffer:*:=!
bfw[298]=       )
bfw[299]=endlocal
bfw[300]=:AppendFileLineToFile-end
bfw[301]=if "[%~4]" NEQ "[]" ( shift & GoTo :AppendFileLineToFile-arg )
bfw[302]=Call :ClearVariablesByPrefix %_AppendFileLineToFile_prefix% _AppendFileLineToFile_prefix & GoTo :EOF

Well, that also looks the same !



I guess now what I need is "ArrayToFile"


I did find something I wrote earlier, but it looks .. too simple

Code: Select all

:ArrayToFile byref InputArray OutputFile
Call :GetArrayParameters %~1 _ArrayToFile_input "" 0
set _ArrayToFile_input 
:ArrayToFile-loop
call echo %%%_ArrayToFile_input%[%_ArrayToFile_input.index%]%%>>%~2
set /a _ArrayToFile_input.index+=1
if %_ArrayToFile_input.index% leq %_ArrayToFile_input.ubound% GoTo :ArrayToFile-loop
GoTo :EOF

::Usage Call :GetArrayParameters InputArray ParameterVariable optional Initialize optional StartIndex=0
:GetArrayParameters
call set _GetArrayParameters.lbound=%%%~1.lbound%%
call set _GetArrayParameters.ubound=%%%~1.ubound%%
call set _GetArrayParameters.count=%%%~1.count%%
call set _GetArrayParameters.index=%%%~1.index%%
if "[%~3]"=="[Initialize]" (
	if "[%_GetArrayParameters.lbound%]"=="[]" set /a _GetArrayParameters.lbound=0
	if "[%_GetArrayParameters.ubound%]"=="[]" set /a _GetArrayParameters.ubound=-1
	if "[%_GetArrayParameters.count%]"=="[]" set /a _GetArrayParameters.count=%_GetArrayParameters.ubound%-%_GetArrayParameters.lbound%+1  2>nul
	if "[%_GetArrayParameters.index%]"=="[]" if "[%~4]"=="[]" set /a "_GetArrayParameters.index=0"
) else (
	if "[%_GetArrayParameters.lbound%]"=="[]" set "_GetArrayParameters_bounds_incomplete=true" & set "_GetArrayParameters_lbound_empty=true"
	if "[%_GetArrayParameters.ubound%]"=="[]" set "_GetArrayParameters_bounds_incomplete=true" & set "_GetArrayParameters_ubound_empty=true"
	if "[%_GetArrayParameters_bounds_incomplete%]"=="[true]" Call :FindArrayBounds %%%~1%% _GetArrayParameters_bounds_incomplete
	if "[%_GetArrayParameters_lbound_empty%]"=="[true]" set "_GetArrayParameters.lbound=%_GetArrayParameters_bounds_incomplete.lbound%"
	if "[%_GetArrayParameters_ubound_empty%]"=="[true]" set "_GetArrayParameters.ubound=%_GetArrayParameters_bounds_incomplete.ubound%"
	if "[%_GetArrayParameters.count%]"=="[]" set /a _GetArrayParameters.count=%_GetArrayParameters.ubound%-%_GetArrayParameters.lbound%+1  2>nul
)
if "[%~4]" NEQ "[]" ( set /a "_GetArrayParameters.index=%~4" 2>nul )
set _GetArrayParameters.name=%~1
set "%~2.lbound=%_GetArrayParameters.lbound%"
set "%~2.ubound=%_GetArrayParameters.ubound%"
set "%~2.count=%_GetArrayParameters.count%"
set "%~2.index=%_GetArrayParameters.index%"
if "[%_GetArrayParameters.name%]" NEQ "" set "%~2=%_GetArrayParameters.name%"
Call :ClearVariablesByPrefix _GetArrayParameters & if "[%_GetArrayParameters.ubound%]"=="[]" ( exit /b 1 ) else ( exit /b 0 )
GoTo :EOF
::GetArrayParameters-END

I gave it a shot anyway

cmd_gBmUBTRATs.png (20.65 KiB) Viewed 13749 times

Unfortunately, :GetLabelRow didn't make it

Code: Select all

::Usage Call :GetLabelRow BatchFile FunctionName optional OutputRow
:GetLabelRow
exit /b 0

Looking at :AppendFileLineToFile, it is also mangled, but differently

Code: Select all

ECHO is off.
::Usage Call :AppendFileLineToFile inputfile outputfile 3 4 50-75 5 6 7 ... N
:AppendFileLineToFile
set "_AppendFileLineToFile_prefix=_AFLTF"
set "_AFLTF_InputFile=%~1"
set "_AFLTF_OutputFile=%~2"
:AppendFileLineToFile-arg
if not defined _AFLTF_Stop set /a _AFLTF_Stop=%_AFLTF_Start%
Setlocal enabledelayedexpansion
if %_AFLTF_Start% GTR 1 set /a "_AFLTF_skip=%_AFLTF_Start%-1"
if %_AFLTF_Start% GTR 1 ( set "_AFLTF_skip=skip^=%_AFLTF_skip%^" ) else ( set "_AFLTF_skip=" )
for /f %_AFLTF_skip% delims= eol= %%a in (' ( type "%_AFLTF_InputFile%" | %SystemRoot%\System32\findstr /N /R /C:".*" ) 2>nul ') do ( 
	for /f "delims=:" %%f in ("%%a") do if %%f GTR %_AFLTF_Stop% GoTo :AppendFileLineToFile-end
	set _AFLTF_buffer=%%a
	if defined _AFLTF_buffer  echo(!_AFLTF_buffer:*:=!
	) 
endlocal
:AppendFileLineToFile-end
ECHO is off.

Everywhere there was an empty line, now says ECHO is off.

Also the 343 line original, is now 286 lines :\

Here is the whole thing

https://pastebin.com/JRia6CKe

[Sponge Belly]

Hi Shodan,

I’m not sure what you’re trying to accomplish, but I suggest you read Calling Functions from a Library, especially Post #4 by Penpen.

HTH!

- SB

[shodan]

Thank you that is some top tier wizardry

In my current design, I planned to mostly rely on %~0 for function switching.

I would achieve that with shortcuts / softlinks and/or hardlink

I am creating a function, that lists all functions in a batch file, then creates/updates/deletes shortcuts/softlink/hardlink to that batch file but each is the name of an internal function.

And I put in as a backup, if the filename is "bfw" to use the first argument as a function call, then shift the arguments. Which shifts the arguments once then GOTO instead of call the destination function.

So far it seems this approach only breaks %* and only when using the ShiftedArgumentCaller function

I will have to think how I can use this new technique to improve on my design !

However for today the question remains
How to expand a variable with poison characters and print it or write it to a file.
I think I used to have this solution, but I cannot find it anywhere ...

thanks !

[shodan]

I have cracked it

Posting before the forum times out again !

This will write the line number and line ranges from inputfile to output file

Code: Select all

::Usage Call :AppendFileLineToFile inputfile outputfile 3 4 50-75 5 6 7 ... N
:AppendFileLineToFile
set "_AppendFileLineToFile_prefix=_AFLTF"
set "_AFLTF_InputFile=%~1"
set "_AFLTF_OutputFile=%~2"
:AppendFileLineToFile-args
for /f "delims=- tokens=1,2" %%a in ("%~3") do ( set "_AFLTF_Start=%%a" & set "_AFLTF_Stop=%%b"  )
if not defined _AFLTF_Stop set /a _AFLTF_Stop=%_AFLTF_Start%
if %_AFLTF_Start% GTR 1 set /a "_AFLTF_skip=%_AFLTF_Start%-1"
if %_AFLTF_Start% GTR 1 ( set "_AFLTF_skip=skip^=%_AFLTF_skip%^" ) else ( set "_AFLTF_skip=" )
for /f %_AFLTF_skip% delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr /N "^" "%_AFLTF_InputFile%"') do ( 
	for /f "delims=:" %%f in ("%%a") do if %%f GTR %_AFLTF_Stop% GoTo :AppendFileLineToFile-end
	set _AFLTF_buffer=%%a
	Setlocal enabledelayedexpansion
	set _AFLTF_buffer=!_AFLTF_buffer:*:=!
	>> "%_AFLTF_OutputFile%" echo(!_AFLTF_buffer!
	endlocal
	) 
:AppendFileLineToFile-end
if "[%~4]" NEQ "[]" ( shift & GoTo :AppendFileLineToFile-arg )
Call :ClearVariablesByPrefix %_AppendFileLineToFile_prefix% _AppendFileLineToFile_prefix & GoTo :EOF

:: Usage Call :ClearVariablesByPrefix myPrefix
:ClearVariablesByPrefix
if "[%~1]" NEQ "[]" for /f "tokens=1 delims==" %%a in ('set "%~1" 2^>nul') do set %%a=
if "[%~2]" NEQ "[]" shift & GoTo :ClearVariablesByPrefix
GoTo :EOF

[miskox]

Very good. Tried and works. But there is a typo:

Code: Select all

:AppendFileLineToFile-args

Missing 'S' at the end:

Code: Select all

if "[%~4]" NEQ "[]" ( shift & GoTo :AppendFileLineToFile-arg )

Saso

[Sponge Belly]

Hi Shodan,

I refer you to Jeb’s answer to SO: How to Read a File.

HTH!

- SB

[shodan]

Thanks,

My current solution is certainly derived from this comment.
I found it in an old version of my functions but the use of echo( and the very similar structure, I'm sure that's where I got it from !


Looking at the differences

He explicitely uses SETLOCAL DisableDelayedExpansion
Which I don't because it's assume that it's not enabled prior.

But adding it would probably make my version more robust.
However, he uses setlocal twice but endlocal only one.
I imagine there needs to be another endlocal after the for loop to restore cmd status to original.

It will be hard to make a FileLinesToArray function with crossing two endlocals !


I'm not sure if I should pipe the output of TYPE into findstr or leave it as is (for some unicode compatibility ?)

I'm not sure what is going on with the usebackq and my version with the escaped for /f parameters.
I've got that from ... somewhere, but I don't know why I'm doing it like that !

Code: Select all

for /f %_AFLTF_skip% delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr /N "^" "%_AFLTF_InputFile%"')
FOR /F "usebackq delims=" %%a in (`"findstr /n ^^ t.txt"`) do (

Last thing

He uses doublequotes to set the buffer variable

Code: Select all

set "var=%%a"

I worried about double quotes in the variable messing that up ? Seem to work both ways however.

Code: Select all

set _AFLTF_buffer=%%a

[Sponge Belly]

Hi Shodan,

There should indeed be a terminating endLocal somewhere after the end of the for /f loop.

Delayed expansion is explicitly disabled to prevent any ^ and ! characters in the line of input from being corrupted. But once the line has been read into the variable, delayed expansion can be safely turned on to allow the line number appended by findStr to be removed, and the line to be echoed on screen verbatim (even if it contains poison characters).

It’s common practice in Batch programming to add double quotes around the definition of a variable (eg, set "var=%%a"). This is to ensure poison characters in the variable’s value don’t cause errors. And it also prevents the accidental inclusion of unwanted spaces. The double quotes will not be included in the variable’s name or its value.

Lastly, useBackq tells the for /f loop to watch out for spaces in filenames. Without it, for /f will treat a filename containing spaces as a string.

HTH!

- SB