Question on Not So basic macros

[Acy Forsythe]

If this works... (and it does)

Code: Select all

FOR /F "tokens=1-26 delims=," %%a IN ('%CMD1%%CMD2%%CMD3%') DO (commands)

Shouldn't this also work? (It doesn't)

Code: Select all

%CMD1%%CMD2%%CMD3%

Or even:

Call %CMD1%%CMD2%%CMD3%

Here is the actual command:

Code: Select all

Set "CMD1=CSVfix read_fixed -f 1:10^,11:32^,57:20^,91:4^,96:25^,125:6 reports\report.rtf"
Set "CMD2= ^| CSVfix trim ^| CSVfix edit -e "s/,/;/g" ^| CSVfix Remove -f 1 -ei "PAGE" ^| CSVfix Remove -f 1 -ei "CUSTOMER""
Set "CMD3= ^| CSVfix eval -e "(len($6^)^)" ^| CSVfix remove -f 7 -e "0" ^| CSVfix exclude -f 7 ^| CSVFix lookup -f 6:1 reports\codes.csv"

I can echo %CMD1%%CMD2%%CMD3% copy/paste and run the command. I can also comment my @echo off and copy/past the expanded and yet failed version of %CMD1%%CMD2%%CMD3% and it also runs fine.

Just trying to execute %CMD1%%CMD2%%CMD3% causes one of the commands to generate a syntax error, but I can't tell which one has the problem and it's difficult to take a step out of the middle and expect the rest of them to work properly so going that route will be difficult.

If I use CALL %CMD1%%CMD2%%CMD3% I get no error message, but nothing happens either. I put a pause after it and all I got was the "press any key to continue..." message.

At this point, I am guessing that there is something I need quoted or something I need to escape that is not necessary inside a FOR /F loop but is necessary if I just want to run the command from the prompt.

Any help would be appreciated...

[Ed Dyreen]

'
You say "Question on basic macros" but it doesn't looks basic to me though.

If you are having problems with complex macros, wouldn't it be easier to start with easier macros first, so you get used to it ?

I find your technique a bit odd, I never place macros as parameters of a for, instead I use parameters of a for like we use parameters in functions.

Are u sure this is what you want ? This is my definition of a macro;
http://www.scriptingpros.com/viewtopic.php?f=129&t=72

I also believe CMD2 &CMD3 are continuations of CMD1, thus cannot be used without the preceding CMD1, I'm confused

CALL %CMD1%%CMD2%%CMD3% does not seem logical, call is only for nested variables, but I don't see any nested variables ?
weird

I would do it like:

Code: Select all

set $command= ^
CSVfix read_fixed -f 1:10,11:32,57:20,91:4,96:25,125:6 reports\report.rtf ^
&CSVfix trim ^
&CSVfix edit -e "s/,/;/g" ^
&CSVfix Remove -f 1 -ei "PAGE" ^
&CSVfix Remove -f 1 -ei "CUSTOMER" ^
&CSVfix eval -e "(len($6))" ^
&CSVfix remove -f 7 -e "0" ^
&CSVfix exclude -f 7 ^
&CSVFix lookup -f 6:1 reports\codes.csv
::
%$command%

[dbenham]

Hi Acy

I agree with Ed, It seems like there would be a much better way of doing whatever it is you are trying to do. Your macro construct is very odd. (I suppose most people that read this will think "Look whose talking " ).

Ed's suggestion looks reasonable, though I think you need to replace all his & with |.

But that does not answer your question. Why doesn't %CMD1%%CMD2%%CMD3% not work directly?

At this point, I am guessing that there is something I need quoted or something I need to escape that is not necessary inside a FOR /F loop but is necessary if I just want to run the command from the prompt.

Actually the answer is the exact opposite

The special characters in the FOR loop need to be escaped, and you have already done that in your definitions of CMD1, 2 and 3. The FOR loop processing consumes the escapes prior to your "DO commands".

But when you try to execute them directly they should not be escaped. Expanding a variable does not consume the escapes, so your command will not execute properly.

This also explains why you can echo the values and then copy and paste to execute - the ECHO command does consume the escapes. Alternatively you could do the following with your current definitions:

Code: Select all

set CMD=%CMD1%%CMD2%%CMD3%
%CMD%

This will work because the SET command will consume the escapes because they are not quoted.

Or you could remove the quotes from your original definitions and then %CMD1%%CMD2%%CMD3% should work directly. But then they will no longer work in the FOR statement.

Dave Benham

[Acy Forsythe]

@Ed Yes it's one command line. I broke it down into 3 sections for readability which was the only goal in this. I created the FOR loop to break down the values.

If you are having problems with complex macros, wouldn't it be easier to start with easier macros first, so you get used to it ?

This is a simple macro, just a string of commands in a variable.

SET "cmd=Echo This is a macro"
%CMD%

Not like I'm trying to create a macro that accepts passed paramaters through a FOR loop and returns complex strings across the endlocal border or anything crazy like that. Just one that contains a piped command string

I think what seems strange to you mostly is the use of the FOR loop, before all this macro business on the forums the FOR loop was used to process the output of commands line by line. My command's output requires processing line by line.

Actually it DID require processing line by line, but then I hit a snag with CSVFix, it won't use piped input for the join command so I was being forced to seperate things into two seperately run command lines. That's when I ran into this problem. I needed CMD1, 2 and 3 to run output to a file, then I needed to run my next set of commands through the FOR loop.

In the end I just added a lookup command to CSVFix which does what I want without breaking the command line chain and I will eventually fix the join command because it's silly that 9 out of 10 commands accept piped input and 5 or 6 don't.


@Dave

I feel a little silly now, I know I've asked some basic questions around here, but that one I knew. I have to stop working with those FOR loop macros and escaping everything

I think you're right about Ed's solution, I forgot about the line continuation which is even easier to read than how I did it.

[dbenham]

One thing to be careful of with line continuation - the ^ at the end of the line enables line continuation AND it escapes the first character of the next line

That is why Ed's code does not have ^ immediately preceding each & (that is really supposed to be |). The special characters need escaping in his code because he has not enclosed his definition in quotes.

Test1 and Test2 are equivalent. Test3 is very different:

Code: Select all

@echo off

echo Test1
echo hello world  ^
|findstr world
echo(

echo Test2
echo hello world^
  ^|findstr world
echo(

echo Test3
echo hello world^
  |findstr world

Output:

Code: Select all

Test1
hello world  |findstr world

Test2
hello world  |findstr world

Test3
hello world

Dave Benham

[Acy Forsythe]

Alright, so unless I'm missing something, Test3 is what I want. Keep in mind, at this point I'm back to a single command string and need to use it in a FOR /F loop...

Code: Select all

Set $Command=^
CSVfix read_fixed -f 1:10^,11:32^,57:20^,91:4^,96:25^,125:6 reports\report.rtf^
 ^| CSVfix trim ^| CSVfix edit -e "s/,/;/g" ^| CSVfix Remove -f 1 -ei "PAGE" ^| CSVfix Remove -f 1 -ei "CUSTOMER"^
 ^| CSVfix eval -e "(len($6))" ^| CSVfix remove -f 7 -e "0" ^| CSVfix exclude -f 7 ^| CSVFix lookup -f 6:1 reports\codes.csv

With that, both of my FOR /F loops look the same when I turn ECHO on. %CMD1%%CMD2%%CMD3% still works, and %$Command% tells me that "| was unexpected at this time"

While doing it this way (once I break the line down further like Ed did) will make the code more readable, I don't see how this is easier or less complicated than... %CMD1%%CMD2%%CMD3%

The only thing I can figure out is the difference between the command strings that show up in the FOR loop when I turn ECHO ON. There is an extra space before the begining pipes on line 2 and 3 that I got rid of and then removed the ^ escaping the two ) But now if I copy/paste right out of the command prompt, my FOR command looks identical, but

[dbenham]

Actually you want option 1 or 2 since the variable is being used in a FOR IN() clause. You escaped all the other pipes, so you need to escape the leading pipe on each line as well. But the line continuation is already escaping the 1st character of the next line. In your case it looks like the next character is a ^, So your leading ^ is escaped and your pipe is not.

I can't be sure this is your problem because this site strips the leading space in code if there is a single leading space. It leaves the leading spaces alone if there are multiple spaces. That is one of the reasons I always indent 2 or more spaces when I indent.

Your code should look like this (line continuation escapes the |):

Code: Select all

Set $Command=^
CSVfix read_fixed -f 1:10^,11:32^,57:20^,91:4^,96:25^,125:6 reports\report.rtf ^
| CSVfix trim ^| CSVfix edit -e "s/,/;/g" ^| CSVfix Remove -f 1 -ei "PAGE" ^| CSVfix Remove -f 1 -ei "CUSTOMER" ^
| CSVfix eval -e "(len($6))" ^| CSVfix remove -f 7 -e "0" ^| CSVfix exclude -f 7 ^| CSVFix lookup -f 6:1 reports\codes.csv

Or like this (line continuation harmlessly escapes a space):

Code: Select all

Set $Command=^
CSVfix read_fixed -f 1:10^,11:32^,57:20^,91:4^,96:25^,125:6 reports\report.rtf^
  ^| CSVfix trim ^| CSVfix edit -e "s/,/;/g" ^| CSVfix Remove -f 1 -ei "PAGE" ^| CSVfix Remove -f 1 -ei "CUSTOMER"^
  ^| CSVfix eval -e "(len($6))" ^| CSVfix remove -f 7 -e "0" ^| CSVfix exclude -f 7 ^| CSVFix lookup -f 6:1 reports\codes.csv

Dave Benham

[Acy Forsythe]

I tried it every way I could try it and it either resulted in executing the first line of my command with an error or giving me the | not expected at this time error, or both.

I just copy/pasted both of your examples and changed my For loop:

From:

Code: Select all

FOR /F "tokens=1-26 delims=," %%a IN ('%CMD1%%CMD2%%CMD3%') DO (

To:

Code: Select all

FOR /F "tokens=1-26 delims=," %%a IN ('%$Command%') DO (

And I get the same parsing error "| was unexpected at this time" I'm really at a loss as to why it doesn't work...

But I did accomplish my goal of making it more readable and still having it work... I broke the command down like Ed suggested but with the Set "CMD=%CMD%MoreCommands" syntax so in the begining, it's almost as easily readable and in the end I'm just passing %CMD% to my FOR loop instead of %CMD1%%CMD2%%CMD3%.

[Ed Dyreen]

'
Why not include the for inside your definition ?

Code: Select all

Set $Command= ^
FOR /F "tokens=1-26 delims=," %%a IN ('%$Commands%') DO ( these things )
::
%$Command%

I assume you build and test the macro one step at a time no ?

When I build a macro, I always include a command at the end, just to be sure.

Code: Select all

Set $Command= ^
these things ^
&echo.This works
::
%$Command%

[Acy Forsythe]

I was testing command by command originally, and everything was working within my FOR loop.

The original problem isn't the problem anymore... The problem is that my SET commands looked like a mess and yours didn't

But I can't for the life of me get it to work with a single SET command, although I did end up breaking it down into 9 lines similar to yours, they just all start with SET and I'm concatenating the command together in a single variable instead of using multiple variables.

So I've accomplished both of my goals and created a new problem. Why doesn't that multi-line SET command work? I did eliminate all but the 1st three commands so a PIPE was included both at the begining and in the middle of the line and still getting the same error.

This works:

Code: Select all

SET "CMD=CSVFix read_fixed -f 1:10^,11:32^,57:20^,91:4^,96:25^,125:6 %WORKDIR%\%PENSRPT%"
SET "CMD=%CMD% ^| CSVFix trim"
SET "CMD=%CMD% ^| CSVFix edit -e "s/,/;/g""
SET "CMD=%CMD% ^| CSVFix Remove -f 1 -ei "PAGE""
SET "CMD=%CMD% ^| CSVFix Remove -f 1 -ei "CUSTOMER""
SET "CMD=%CMD% ^| CSVFix eval -e "(len($6^)^)""
SET "CMD=%CMD% ^| CSVFix remove -f 7 -e "0""
SET "CMD=%CMD% ^| CSVFix exclude -f 7"

...
FOR /F "tokens=1-26 delims=," %%a IN ('%CMD%') DO (parse each line)

The FOR loop is necessary because it can do something CSVFix cannot, save information from the previous line and handle the next line(s) accordingly. That's next on my list of enhancements as soon as I finish this script.

[Acy Forsythe]

Ok, so my Batch just reached 250 lines and I'm only about halfway through it. It takes roughly about 10 minutes to run at this point and I though I thought I might be able to pick up some speed here, but I'm getting an error.

Code: Select all

SET LF=^


::Above 2 blank lines are required - do not remove

SET ^"\n=^^^%LF%%LF%^%LF%%LF%^^"

SET macro_Call=FOR /f "tokens=1-26" %%a IN


SET macro.GetTime=DO (%\n%
  setlocal enableDelayedExpansion%\n%
  SET "t=0"%\n%
  FOR /F "tokens=1-4 delims=:." %%A IN ("!time: =0!") DO SET /a "t=(((1%%A*60)+1%%B)*60+1%%C)*100+1%%D-36610100"%\n%
  FOR %%v IN (!t!) DO ENDLOCAL^&IF "%%~a" neq "" (set "%%~a=%%v") ELSE ECHO:%%v%\n%
)


SET macro.DiffTime=DO (%\n%
  setlocal enableDelayedExpansion%\n%
  SET /a "DD=(%%~b)-(%%~a)"%\n%
  IF !DD! lss 0 SET /a "DD+=24*60*60*100"%\n%
  SET /a "HH=DD/360000, DD-=HH*360000, MM=DD/6000, DD-=MM*6000, SS=DD/100, DD-=SS*100"%\n%
  IF "!HH:~1!"=="" set "HH=0!HH!"%\n%
  IF "!MM:~1!"=="" set "MM=0!MM!"%\n%
  IF "!SS:~1!"=="" set "SS=0!SS!"%\n%
  IF "!DD:~1!"=="" set "DD=0!DD!"%\n%
  FOR %%v IN (!HH!:!MM!:!SS!.!DD!) DO ENDLOCAL^&IF "%%~c" neq "" (set "%%~c=%%v") ELSE ECHO:%%v%\n%
)


SET macro.ProcessRPT=DO (%\n%
    setlocal enableDelayedExpansion%\n%
    SET "CMD=CSVfix read_fixed -f 1:10^,11:32^,57:20^,91:4^,96:25^,124:6 %WORKDIR%\%%~a"%\n%
    SET "CMD=%CMD% ^| CSVfix trim"%\n%
    SET "CMD=%CMD% ^| CSVfix edit -e "s/,/;/g""%\n%
    SET "CMD=%CMD% ^| CSVfix Remove -f 4 -ei "CODE""%\n%
    SET "CMD=%CMD% ^| CSVfix eval -e "(len($4^)^)""%\n%
    SET "CMD=%CMD% ^| CSVfix remove -f 7 -e "0""%\n%
    SET "CMD=%CMD% ^| CSVfix exclude -f 7"%\n%
    FOR /F "tokens=1-7 delims=," %%A IN ('!CMD!') DO (%\n%
    IF %%A=="" (%\n%
    SET "AN=!Acct!"%\n%
    SET "NM=!Name!"%\n%
    ) ELSE (%\n%
    SET "AN=%%~A"%\n%
    SET "Acct=%%~A"%\n%
    SET "NM=%%~B"%\n%
    SET "Name=%%~B"%\n%
    )%\n%
    SET LINKED=""%\n%
    SET "INCMD=CSVfix find -f 1 -e !AN! %WORKDIR%\%%~b"%\n%
    SET "INCMD=!INCMD! ^| CSVfix edit -e "s/,/;/g""%\n%
    FOR /F "tokens=1-5 delims=," %%W IN ('!INCMD!') DO (%\n%
    SET "LINKED=%%~Y"%\n%
    )%\n%
    IF !LINKED!=="" (%\n%
    SET "LINKED=!AN!"%\n%
    )%\n%
    ECHO:!AN!,!NM!,%%~c,%%~d,%%~e,%%~f,!LINKED!%\n%
    )%\n%
)


REM calling my macro -
%macro_call% ("%GNVRPT% %GNVLINKED%") %macro.ProcessRPT% >> %WORKDIR%\temp.csv

GNVRPT and GNVLINKED are filenames with no spaces. WORKDIR also contains no spaces and is a direct subdirectory.

The error I get is :

)"" was unexpected at this time.

I beleive that it gets past the %CMD% part but I can't tell for sure.

[Acy Forsythe]

I figured it out. I needed to double-up the ^ before the parenthesis. The rest of the ^ were not eaten up by the set command, but those were.

I think I have a handle on this macro thing now

I still don't fully understand the %\n% but I know where to look for the explanation of it.

[Acy Forsythe]

For purposes of understanding, all that CMD stuff in my macro can be replaced by passing a csv file through the FOR loop with the TYPE command like:

FOR .... IN ('TYPE mycsvfile.csv') DO ...

And in fact, that's all the output of the CSVFix command is doing is outputing line by line a CSV file, no different than TYPE would do. The only reason I'm using CSVFix is because it was more efficient than writing a batch file to parse a tab delimited report file that had headings, page numbers, column headings etc... and outputing a CSV file.

But Ed got me side-tracked and so in the end I did not increase the read-ability of my script, but I did increase my macro understanding.

So now I got the hang of creating a complex macro that does something, and just about 5 seconds ago had a mental breakthru and understand how your passing values back OUT of your macros (the FOR %%v IN ....) but I don't understand some things and it's probably because I've been too busy to spend any time trying to understand them...

Like the %\n% I know why it's needed. I do not know why just a ^ line continuation character wouldn't work. Or maybe a couple of them.

I seem to remember in soome off your earlier macros you actually DID use just a single ^ and it worked...

I also don't understand why Set LF=^ requires two blank lines under it. I do understand that we are actually setting LF = (I think) just a CR/LF and that the ^ is eaten up right? I hope that's right or I'll have to start over. Anyway if it is right, what's the 2nd blank line for?

Also, I get what ForEntireLine is used for now, incase your ... IN (whatever) ... needs to have EOL and Delims disabled so the whole thing goes into %%a or in your examples, %%v. Again if that's not right, I may as well call it quits. But the part I don't understand, is why that needs to be:

Code: Select all

^^^^^^^"eol^^^^=^^^^^^^%LF%%LF%^%LF%%LF%^^^%LF%%LF%^%LF%%LF%^^^^ delims^^^^=^^^^^^^"

I've read the related threads where you and Jeb discussed it, but I still don't get it. Anyway, that's where my understanding lies at this point. I'm going to have to start writing all of this down somewhere...

EDIT...

WAIT!

One thing to be careful of with line continuation - the ^ at the end of the line enables line continuation AND it escapes the first character of the next line

This answers part of one of my questions (actually it just corrects the part I misunderstood about it) The LF=^ isn't being set to a CR/LF, the ^ gets rid of the CR and we're left with the LF ? But still doesn't explain the extra line... I'm going to go read that thread again

After I take a nap, this is exhausting :p

[dbenham]

Congrats on your 1st complex macro

Like the %\n% I know why it's needed. I do not know why just a ^ line continuation character wouldn't work. Or maybe a couple of them.

I seem to remember in soome off your earlier macros you actually DID use just a single ^ and it worked...

Yes, line continuation by itself works, but then the entire macro is defined as one single line of text - This is very difficult to read and debug.

It was jeb's idea to introduce line feeds into the actual definition, and it makes debugging much easier.

I also don't understand why Set LF=^ requires two blank lines under it. I do understand that we are actually setting LF = (I think) just a CR/LF and that the ^ is eaten up right? I hope that's right or I'll have to start over. Anyway if it is right, what's the 2nd blank line for?
...
... This answers part of one of my questions (actually it just corrects the part I misunderstood about it) The LF=^ isn't being set to a CR/LF, the ^ gets rid of the CR and we're left with the LF ?

I'm not sure I have this completely right, but I'll give it a go.

The <CR> is always stripped by the parser, so it would never be included in the variable definition with a construct like this. That being said, jeb has a clever trick to set a CR variable, totally unrelated to this discussion.

Now let's take it step by step.

Code: Select all

set TEST=^
HELLO

Result: TEST=HELLO
No need to say more

Code: Select all

set test=^

Hello

Result: TEST=<LF>HELLO
The ^ introduces continuation and escapes the 1st character on the next line. The 1st character is <LF> (the <CR> has been stripped). Because the <LF> is escaped it does not terminate the definition and it is included in the definition. Since we are not terminated yet, the next line containing HELLO is also included in the definition. The CR/LF at the end of this line finally terminates the definition.

Code: Select all

set test=^


HELLO

Result: TEST=<LF>
The ^ introduces continuation and escapes the 1st character on the next line. The 1st character is <LF> (the <CR> has been stripped). Because the <LF> is escaped it does not terminate the definition and it is included in the definition. The parser looks for additional characters on the next line, but the very 1st thing it sees is CR/LF which terminates our definition. We have our desired result.

Also, I get what ForEntireLine is used for now, incase your ... IN (whatever) ... needs to have EOL and Delims disabled so the whole thing goes into %%a or in your examples, %%v. Again if that's not right, I may as well call it quits. But the part I don't understand, is why that needs to be:

Code: Select all

^^^^^^^"eol^^^^=^^^^^^^%LF%%LF%^%LF%%LF%^^^%LF%%LF%^%LF%%LF%^^^^ delims^^^^=^^^^^^^"

No worries Acy - You got it right

The native syntax without using a macro is odd enough. The syntax then had to be escaped twice so that it could be used in a macro definition. The act of defining the macro consumes one set of escapes. The act of executing the macro consumes the second set. The syntax for escaping the <LF> is not at all intuitive, but it does make sense (after staring at it many times). I can now do my own <LF> escape sequencing, but I am still not very fast. Normally I just copy and paste from existing code.

It helps to set a variable VAR1 = to the above, then look at the definition using SET VAR1. Then set VAR2=%VAR1% and look at the definition of VAR2.

Now think about the keystrokes it takes to define <LF>, and work in reverse to go from the definition of VAR2 back to the syntax that defines VAR1. That's kind of the process I went through anyway.

Hope it helps

Dave

[Acy Forsythe]

Thanks Dave, that helped. So the \n is not functionally necessary, but since it's difficult to trouble-shoot macros without using the SET command, it makes them easier to read and find errors etc...

And eol and delims end up just being = %LF% ?

When I use the example you provided, it looks like they are both = to the same thing, but the definition is what throws me off.

Delims = ^^^^^^^
Eol = ^^^^^^^%LF%%LF%^%LF%%LF%^^^%LF%%LF%^%LF%%LF%^^^^

EDIT: Nevermind, typing it in like that showed me the light I think. They are both = ^^^^^^^ and the rest is to seperate them kind of like the two blank lines used to define LF.