Comments without increasing macro size

[penpen]

I wonder, whether i see it right or not:
Don't you increase the macro size by 5 characters per line ("<nul " in "^\n\n<nul ^")?

penpen

[jeb]

I wonder, whether i see it right or not:
Don't you increase the macro size by 5 characters per line ("<nul " in "^\n\n<nul ^")?

No the trick is, that the <nul is related to the SET command, not the the macro.
So it's not present in the macro.

So the same is true here, the <nul is only used in the building process of the macro

Nevertheless the %\n% may induce an unwanted "<nul" at the end of the macro which may be suboptimal for such macro usage (here macro $readLine):
Code:
(
%$readLine% PARAMS
) < file

Code: Select all

@echo off
setlocal enableDelayedExpansion
set LF=^


set ^"\n=^^^%LF%%LF%^%LF%%LF%^<nul ^^"
echo on
set readlineMacro=^<file.txt (%\n%
set /p input=%\n%
)

@echo off
echo(
set readlin

C:\temp>set readlineMacro=<file.txt (
set /p input=
) 0<nul

readlineMacro=<file.txt (
set /p input=
)

You can see the difference here between the definition of the macro and the content of the macro

jeb

[penpen]

I haven't realized the trick with "<nul".

But i think i have found out where the "<<nul" comes from in the case above,
when i was trying to better understand the trick; i have externalized the "<nul" from variable "\n", but it should be (except one additional space) (nearly) the same:

Code: Select all

@echo off
cls
setlocal enableDelayedExpansion
set LF=^


set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
set "\\=<nul ^"

echo on
set macro=^<file.txt (%\n% <nul ^
%=comment line1%<nul ^
set /p input=%\n%
echo.one_token)

This is the result (on my win xp home):

Code: Select all

Z:\>set macro=<file.txt (
  /p input=
echo.one_token) 0<set

The "<nul" (from the second comment) is placed where the "set" can be seen.

But i have no idea why this happens.

penpen

Edit: Adding a space in front of variable "\\" seems to solve the problems:

Code: Select all

set "\\= <nul ^"

Edit2: But then you add additional spaces into the macro.

[penpen]

This seems to work, without increasing the macro size: (Edited: I was wrong.)

Code: Select all

@echo off
cls
setlocal enableDelayedExpansion
set LF=^


set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
for /f %%a in ('"prompt $H&for %%b in (1) do rem"') do set "\\= %%a^"

echo on
set macro=(echo Line1%\n%
%=comment line1%%\\%
%=comment line2%%\\%
%\\:)=comment line3%
%\\:)=comment line4%
echo.oneToken%)
@echo off

echo(
set macro

endlocal
goto :eof

penpen

Edit: Added an alternate comment format using only one variable.
Edit2: Found out, that "<nul " is not necessary in the above solution, so i removed it.
Edit3: Liviu was right, that this is "just a screen illusion".

[jeb]

I made some tests today to get behind the mechanics of the "<nul ^".

If found 2 solutions for the macro comments:

Code: Select all

@echo off
setlocal
prompt [PARSER]
set LF=^


set "\\=<nul SETLOCAL^"

@echo on
echo Hello%\\%
%=comment1%%\\%
%=comment2%%\\%
world

Ok, I must admit that the word SETLOCAL has absolute no meaning
you can use any single token, but not an empty one.

The second solution is embrassed simple.

Code: Select all

@echo off
setlocal
prompt [PARSER]
set LF=^


set "\\=< nul ^"

@echo on
echo Hello%\\%
%=comment1%%\\%
%=comment2%%\\%
world

It's only necessary to add a space between "<" and "nul".

I can not explain the complete behaviour of "<nul ^", but it has an effect of the processing of the next line.
In spite of the normal multiline caret, this form doesn't escape the first character of the next line.
But when the first charcater of the next line is a redirection character it consumes one token before OR after the multiline caret

In this sample I add colons to the "NUL", that doesn't change anything, but you can see which NUL: is used by counting the colons.

Code: Select all

@prompt [PARSER]
@echo on
echo Test1<nul: invisble^
<nul:: world

@echo ------------
echo Test2<nul: ^
<nul:: world

[PARSER]echo Test1 world 0<nul::
Test1 world
------------

[PARSER]echo Test2 0<world
Test2

The second test is impressive , the redirection uses the file "world" instead of any NUL, it's because the parser shuffle the line to:

Code: Select all

[ParserView]echo Test2<NUL: NUL::^
< world

The word before the multiline caret is removed and the first redirection is overridden by the second one.

For me it's barely comprehensible how the source code of the parser could look like

jeb

[penpen]

Another interesting parser behavior (only added a space prior ^ to Test1):

Code: Select all

echo Test1<nul: invisble ^
<nul:: world

Result:

Code: Select all

Z:\>echo Test1 invisble <nul:: world 0<nul:

penpen

[jeb]

Another interesting parser behavior (only added a space prior ^ to Test1):
Code:
echo Test1<nul: invisble ^
<nul:: world

This breaks the "<nul ^" trick, as the "<nul" has to be placed directly before the multiline caret (or multiline caret with a prefixed token).

So now the multiline caret works as always, it escapes the first character of the next line.
Only the first "<nul:" works here as a redirection.

So the output of your command is

Test1 invisble <nul:: world

But your test has inspired me to retest another rule with a suprising result:
I tested if there is any difference between a multiline caret and a "natural" linefeed or a linefeed from a percent expansion.
I never saw a difference until now

Code: Select all

set LF=^


echo Test1a_ prefix^
_Line2

echo Test1b_ prefix^%LF%_Line2

@echo(

echo Test2a_<nul: prefix^
_Line2

echo Test2b_<nul: prefix^%LF%_Line2

Test1a_ prefix_Line2
Test1b_ prefix_Line2

Test2a__Line2
Test2b_ prefix_Line2

Both Test1 cases works the same way as expected.

But in Test2b the prefix isn't removed anymore
It's get more and more curious ...

jeb

[Liviu]

For me it's barely comprehensible how the source code of the parser could look like

I tested if there is any difference between a multiline caret and a "natural" linefeed or a linefeed from a percent expansion. I never saw a difference until now

Quite odd, indeed. Seems like the next token after redirection is given preferential treatment in the early parsing phases. Doesn't make any sense, unless there was some special syntax considered at one time, either undocumented or dropped since, but which left footprints in the code. Or maybe it's just a plain dumb bug.

Liviu

P.S. Couple of late followups...

Edit2: But then you add additional spaces into the macro

The original \n plus ^^ for comments only add one space per line, too.

This seems to work, without increasing the macro size

No, that's just a screen illusion. If you redirect the output to a file, you'll see that SPACE+BACKSPACE are actually part of the macro body.

[penpen]

I think, i've found a way, to comment even single arguments of a command within a macro, although i'm not sure if this is needed often...:

Code: Select all

@cls
@setlocal
@set LF=^


@set ^"\n=^^^%LF%%LF%^%LF%%LF%^<nul ^^"

:: adn := argument description newline single params
:: ead := end of argument description
@set ^"\adn=^<nul ^^^%LF%%LF%^<nul ^^"
@set ^"\ead=nul^^^%LF%%LF%^%LF%%LF%^<nul ^^^%LF%%LF%^%LF%%LF%^<nul ^^"


set test=(cmd%\adn%
descr1 arg1%\adn%
descr2 arg2%\adn%
descr3 arg3%\adn%
 "(<-- thespace in front of the quotation marks in needed) Some usefull description of arg4 with: &()[]{}^=;!'+,`~ " arg4%\adn%
%\ead%
command2 arg2^)

@endlocal
@goto :eof

At least this works on win xp and should output this:

Code: Select all

Z:\>set test=(cmd arg1 arg2 arg3 arg4
command2 arg2) 0<nul

penpen

Edit: Currently this adds a single space after the last token in the commented line.

[penpen]

I found out, that my method above also could be used to create "normal" comments (not only for single arguments).
For better readability i used variables "[" and "]", which are the same as "\adn" and "\ead".
To start a comment just use the variable "[" instead of the variable "\n"
and end the comment lines with the variable "]" as the only token in that line.
If you use a single token comment it has to start on the first character in that line immediately followed by a ^ as the last character in the line.
If want to use more token, then you have to encapsulate them in doublequotes, and the first character in the line has to be a space followed by the enquoted comment, followed by the hyphen as the last character in that line.
Within the comment (under XP) only these characters are forbidden: 0x00 (NUL), 0x0A (NL), 0x1A (SUB), 0x22 (").
You have to add a space as a line first character and a ^ at the end of the comment line.
Please test if that works in other systems, too (Tested: win xp).

Code: Select all

@echo off
cls
setlocal
set LF=^


set ^"\n=^^^%LF%%LF%^%LF%%LF%^<nul ^^"

:: adn := argument description newline single params
:: ead := end of argument description
set ^"\adn=^<nul ^^^%LF%%LF%^<nul ^^"
set ^"\ead=nul^^^%LF%%LF%^%LF%%LF%^<nul ^^^%LF%%LF%^%LF%%LF%^<nul ^^"

:: only to make it more readable i created variables [ and ]: [ == \adn, [ == \ead
set ^"[=^<nul ^^^%LF%%LF%^<nul ^^"
set ^"]=nul^^^%LF%%LF%^%LF%%LF%^<nul ^^^%LF%%LF%^%LF%%LF%^<nul ^^"

@echo on
set macro=^(cmd1 arg11 arg21 arg31 arg41%\n%
cmd2 arg21 arg22%[%
single_Token_Comment^
 " ... !#$%&'()*+,-./0123456789:;<=>? ... 0xFF without 0x00, 0x0A, 0x1A, 0x22"^
%]%
cmd3 arg31 arg32 arg33%\n%
^)
@echo off

endlocal
goto :eof

The result should look like this:

Code: Select all

Z:\>set macro=(cmd1 arg11 arg21 arg31 arg41
cmd2 arg21 arg22
cmd3 arg31 arg32 arg33
) 0<nul

penpen
Edit: Adds a space per comment block to the macro; a better version could be found on my next post.

[penpen]

Finally, i got my variation of documentation to work without any extra character:
- insert a line containing "%[%" to start a comment, and
- insert a line containing "%]%" to end a comment
- a multilinecomment must be a single token: per line a token immediately followed by an "^" at the end of the line, or
- the comment may be a single space, or single/multiple ":" followed by a single string, immediately followed by an "^" at the end of that line.
The string is allowed to contain any character except: 0x00 (NUL), 0x0A (NL), 0x1A (SUB), 0x22 (").
You could also use this method to comment single parameters (as this method is an advanced method of this, see my upper posts):

Code: Select all

@echo off
cls
setlocal
set LF=^


setlocal disableDelayedExpansion
set ^"\n=^^^%LF%%LF%^%LF%%LF%^<nul ^^^"
set ^"[=%%~#^<nul ^^^%LF%%LF%^<nul ^^^"
set ^"]=^<nul^^^%LF%%LF%^%LF%%LF%^<nul ^^^"
endlocal & (
@echo on

for %%# in ("") do set macro=^(cmd1 arg11 arg21 arg31 arg41%\n%
cmd2%[%___comment_cmd2_______%]%
 arg 1%[% "comment argument 1 is no numeric equivalent of a handle"%]%
 arg 2%[%_"comment argument 2 is no numeric equivalent of a handle"%]%
 arg3 arg4 arg5%\n%
%[%
comment_1^
 "comment 2"^
::"comment 3 chars in {0x00, ..., 0xFF} \ {0x00, 0x0A, 0x1A, 0x22}"^
%]%
cmd3 arg31 arg32 arg33%\n%
^)

@echo off
)

set macro > macro.txt
endlocal
goto :eof

Tested on WinXP.

penpen

Edit1:
Changed the text above a little bit and added the single parameter commenting.
Added a tilde character prior to the last doublequte in the definition of the environment variables "\n", "[", and "]" to avoid unfinished strings (could be in problem in macros that define macros).
Added the expansion of nothing (for %%#, %%~#) to avoid misinterpretations of single digits as a numeric equivalent of a handle.
Added the single argument comments (cmd2).
Encapsulated the environment variables "\n", "[", and "]" within setlocal blocks to not corrupt these variables within an actual environment.

[dbenham]

You might want to define the term "single string". The string must be quoted if it contains any token delimiter like space, comma, equal, etc. You obviously did that, but the definition is not clear.

Unlikely to be an issue, but the total length of the comment must be < 8191 characters.


Dave Benham

[Liviu]

I tested if there is any difference between a multiline caret and a "natural" linefeed or a linefeed from a percent expansion. I never saw a difference until now

I thought for a moment that the difference could be in the hidden CRs. A "natural" linefeed is actually a CR+LF on disk, and if the parser somehow worked on the raw binary stream before the text conversion which translates CR+LF -> LF then it could matter. Of course, it would make little sense for any parser to do that, but then we are talking about 'cmd' here However, I don't think that's the case. Can't fully disprove it, since it doesn't seem to be possible to insert a literal %CR% during early expansion, but I tried to hack a batch file so that it used just LF line endings (not CR+LF), and the difference still showed.

Back on topic, I don't think we have a final resolution on newlines and single-token continuations at this point.

I suppose it's a 90% solution.

Actually, I think it could be a (close to?) 100% solution, just with slightly different "operating instructions".

Code: Select all

@echo off & setlocal

set LF=^


set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
set ^"//=^^^%LF%%LF%^%LF%%LF%^<nul ^^"

:: code line before a full line %=..=% comment must end in '%//%'
:: %=..=% comment line followed by another %=..=% comment must end in '^^'
:: %=..=% comment line followed by code may end in '^^', or have no ending

set ^"var-with-comments=line_1%\n%
line 2%//%
%= comment_2 =%
line_3%//%
%= comment_3a =%^^
%= comment_3b =%
line 4%//%
%= comment_4a =%^^
%= comment_4b =%^^
%= comment_4c =%^^
line_5^"

set "var-with-comments"

endlocal & goto :eof

Code: Select all

C:\tmp>macro-full-line-comments
var-with-comments=line_1
line 2
line_3
line 4
line_5

The above seems to work without carrying any extras in the macro expansion.

Liviu

[Ed Dyreen]

I prefer penpen's solution, so I can just use \n without any conditions.

Code: Select all

@echo off &prompt $g &set $lf=^


::
set ^"$n1c=^^^%$lf%%$lf%^%$lf%%$lf%^<nul ^^"
set ^"$c1=%%~#^<nul ^^"

%=   =%for %%# in ("") do set ^"test=for %%? in (1,2) do if %%?==2 (%$c1%
%$c1%%=           this is a multi line comment, next can be data or a multi line comment=%
%$c1%%=           this is a multi line comment, next can be data or a multi line comment=%
%$c1%%=           this is a multi line comment, next can be data or a multi line comment=%
%=      =%echo.it works :^^^)%$n1c%
%$c1%
%=   =%)else set $="

( %test% )

set test

pause
exit

Code: Select all

it works :)
test=for %? in (1,2) do if %?==2 (echo.it works :^)
)else set $=
Druk op een toets om door te gaan. . .

[Liviu]

I prefer penpen's solution, so I can just use \n without any conditions.

That's a nifty craft of penpen's, no argument there. But if you want to "just use \n without any conditions" then you have to append that '^<nul ' to the '\n' macro definition itself. I can't say I followed every bit and lead, so I may well be missing something, but I don't know that it's been established beyond doubt that there is no construct or context where that extra '^<nul ' could possibly introduce some unexpected/different behaviors. In fact, you were the first one to point such a suspect case http://www.dostips.com/forum/viewtopic.php?p=32592#p32592 earlier in this very thread.

The alternative would be to use the traditional '\n' by default, then define a new macro with '^<nul ' appended, and only use it before full-line comments. That's a very specific use-case, and the behavior of '^<nul ' in that particular context has been discussed, dissected, and looks to be pretty well understood - so there is far less chance of surprises or misbehaviors that way. But in that case, you need for one line of the macro to use different endings depending on whether the next line is a full-line comment or not. And at that point, it's no longer that much different from my code posted above.

To sum it up, I don't like the code I posted because the line ending of one line depends on what comes on the next line - whether it's a full-line comment or not - which makes it tedious and error-prone to copy/paste/delete/move full-line comments in a macro. However, I am also not entirely sold on the idea of hacking the tried-and-true '\n' for what is essentially a rare usage case. In the end, it comes down to personal preference, I guess.

Liviu