Makeit, scriptable make engine for DOS

[Kochise]

Hi, new to the forum, just wanted to seek for help for my little batch file here :

https://github.com/Kochise/makeit

Basically it reads text files, can include some more like russian dolls, then interpret the resulting script to execute the selected executable on selected files (filtered by extension, excluded by part of path/file name). The principle is to replace make from cygwin/msys with something lighter (50 KB) and that is multiprocessing aware. Yes, DOS is.

Currently I have some features lacking, see the todo list :

1- Can bear other code pages (can execute the bat file with utf8 accents and the string won't be transformed into mess)
2- Can cope arbitrary number of input arguments (for the intermediate batch file execution and error feedback)
3- Synchronize output to follow the input order (must probably be a simple buffering into a text file to flush on certain event)
4- Find a way to parse quoted strings/paths correctly without loosing count of quotes
5- Someone to explain me clearly how the delayed expansion really works, and why it is a good idea (or point me the coder responsible to shot him in the head for that hair tearing crap)
6- Speed execution, especially the 'Resolving' parts during the initialization phase

I'm not an expert of batch files, yet I think I already reached something working and valuable. Once you figured the logic behind it, you can then almost automate everything. This is no cron job, this is no programming language, this is just a make engine.

Hope it can help you, and you can help me in return to improve it.

Kochise

[penpen]

You might create some (standalone and minimized) examples to show the issues you want to be solved one by one.

It think it is hard to find your issues in your code; for example:
How do you ever get an issue with utf-8 if you never change codepage?
Do you read from utf-8 encoded files, or do you execute "makeit.bat" under utf-8 and store the result (somehow) into another codepage (eg. piping the character data to another cmd.instance using another codepage)?

It is also hard to find the code sections you are referring to... (for example "'Resolving' parts during the initialization phase"); it might be usefull to just give the line numbers.

Ad 5) This might help:
https://stackoverflow.com/questions/4094699/how-does-the-windows-command-interpreter-cmd-exe-parse-scripts/4095133#4095133
Using delayed Expansion has multiple advantages, for example see:

Code: Select all

@echo off
setlocal enableExtensions enableDelayedExpansion
set "test=Echo this & echo that"
echo !test!
:: versus
echo %test%
goto :eof

penpen

[Kochise]

Hi, thanks for the feedback. I sure would have provided an example, some are in the 'makefiles' and 'ImageMagick' folders. Just run the batch files there, they'll call makeit.bat with additional parameters.

1- Code page : when you have an utf8 text file as input script, if there are strings with accent, they are not converted into the 'cmd' equivalent, so you often get things wrong. It requires to stick to ascii characters.

2- Line 153, I emit a command line executor to be called by the scheduler somewhere later in the file. Currently it is limited to 10 parameters, but I know there is a better way to transmit a whole line of parameter in one shot without the risk of having, say, quoted strings separated into several parameters.

3- Line 1136, after the execution of one command line, emit the generated output into the main stream. Synchronization of output would require to store the output temporarily and emit it only following the order of the listed files (line 112).

4- Line 1203, the parsing of quoted strings gets exploded into separate arguments

6- Line 179, recursive file inclusion ; line 285, recursive static variable resolution. This... is... damn... slow... Most notably because I separate into two files the lines with static parameters (or identified as such) and those which don't. I resolve one file line by line, then merge the two back into one, using the line number as reference to get the resolved script in the same order than the original script.

5- The problem is to keep track of variable content going from %% to !! then back, the behavior remains cryptic to me. I got it right more by trial and error than real understanding of the underlying inherent processing taking place.

Hope it clears some misunderstanding.

[penpen]

I currently have only few time, but this should help you on task 1:

Code: Select all

@echo off
setlocal enableExtensions enableDelayedExpansion
:: get current codepage, if not available assume cp 850
set "cp=850"
for /f "tokens=2 delims=:." %%a in ('chcp') do set "cp=%%~a"

:: load data from file with bom
>nul chcp 65001
< "test.utf8.bom.txt" (
   rem consume bom
   >nul pause
   >nul pause
   >nul pause
   for /f "tokens=* delims=" %%a in ('findstr "^"') do echo(#%%~a
)

:: load data from file without bom
>nul chcp 65001
< "test.utf8.txt" (
   for /f "tokens=* delims=" %%a in ('findstr "^"') do echo(#%%~a
)


::restore codepage
>nul chcp %cp%
goto :eof

Exemplary file content with and without bom:

Code: Select all

äöü

In hexadecimal:
- "test.utf8.bom.txt": "EF BB BF C3 A4 C3 B6 C3 BC"
- "test.utf8.txt": "C3 A4 C3 B6 C3 BC"

penpen

[PaperTronics]

5- Someone to explain me clearly how the delayed expansion really works, and why it is a good idea (or point me the coder responsible to shot him in the head for that hair tearing crap)

Because you're new to Batch Programming, I'm gonna make it clear to you in a very simple way.

Delayed Expansion has been a hard command to understand for beginners, even I, when I was a beginner wondered why the hell does every "great" batch program contain the line "Setlocal EnableDelayedExpansion", but then I finally, understood what Delayed Expansion did.

Basically,Delayed Expansion enables you to write the variables surrounded by "!" exclamation marks. If you use % signs to use a variable, CMD opens it at the reading time, which causes all the common problems. If you use ! marks to use variables then all the variables that you define in your code are opened by CMD at executing time.

Quite difficult to understand, isn't it?


So let's take an example:

Code:

Code: Select all

@echo off
cls
title Print first 10 Natural numbers...

:: setting up a varibale which we will use to print first 10 natural numbers...
set counter=1
echo THe value of counter variable [Before loop]: %counter%
echo.
echo.
echo.
echo THe value of counter variable [Inside loop]: %counter%
for /l %%a in (1,1,10) do (
   rem Printing numbers using standard syntax of variable usage...
   echo %counter%
   set /a counter=%counter%+1
   )

:: Now as per logic... cmd should print 1, then increment the value of variable counter... then printing 2,then....so on upto 10.
echo.
echo.
echo.
echo The value of counter variable [After loop]: %counter%

echo.
pause
exit /b

Run the code above. The output should be similar to this:

Code: Select all

THe value of counter variable [Before loop]: 1



THe value of counter variable [Inside loop]: 1
1
1
1
1
1
1
1
1
1
1



The value of counter variable [After loop]: 2

Press any key to continue . . .

As you just saw, the counter variable didn't change at all, it's value stayed 1 all along in the loop and after the loop it changed to 2. All because you were too lazy to write one extra line of code and use ! marks instead of % marks your whole code is now messed up.


Using the delayed Expansion:

Code: Select all

@echo off
cls

::making all the following variables local and enabling the ! ...
setlocal enabledelayedexpansion

title Print first 10 Natural numbers...

:: setting up a varibale which we will use to print first 10 natural numbers...
set counter=1


::Note: you can simultaneously use both the syntax,but use only ! var. inside the for loop...

echo THe value of counter variable [Before loop]: %counter%
echo.
echo.
echo.
echo THe value of counter variable [Inside loop]: %counter%
for /l %%a in (1,1,10) do (
   rem Printing numbers using standard syntax of variable usage...
   echo !counter!
   set /a counter=!counter!+1
   )

:: Now as per logic... cmd should print 1, then increment the value of variable counter... then printing 2,then....so on upto 10.
echo.
echo.
echo.
echo THe value of counter variable [After loop]: %counter%

Output:

Code: Select all

THe value of counter variable [Before loop]: 1



THe value of counter variable [Inside loop]: 1
1
2
3
4
5
6
7
8
9
10



THe value of counter variable [After loop]: 11

Tada!!!! The problem is now successfully solved! I think you've got it!


If you haven't understood or want to know some more effects and defects of Delayed Expansion then you can visit this link: http://www.thebateam.org/2017/03/how-to ... local.html (right-click, open link in new tab)

(Now someone else jumps in and says "OH! you're ust giving this link to increase your websites views" to which I'll reply "I've just offered it, dumbo!")




PaperTronics

[Kochise]

Batch copying lisp macros... #shakeshead

[ShadowThief]

If there's one thing that Microsoft is really good at, it's ensuring backwards compatibility (seriously, check out this video of somebody upgrading through every Microsoft OS between DOS 5 and Windows 7).

Delayed expansion is disabled by default because it didn't exist in older versions of batch, and it's super important that that script you wrote in Windows 3.1 still works in Windows XP (although not the 64-bit version of XP if your script uses the DEBUG command).

[penpen]

Just run the batch files there, they'll call makeit.bat with additional parameters.

It would be better, if you just give the batch where things go wrong, instead that one have to guess, what might be an unwanted result in your eyes, after worked into that (huge) batch.
Because of that you should create minimal examples (with probably <= 30 lines per demo batch files), that demonstrates what is going wrong.

Nevertheless:
Ad 2)
I don't see the beenefit to use such a batch file (created at 150-156):
You instead could directly echo line.

The best you could do, if you want to use an external batch is to store the line in a variable, call the batch file with the variable, and echo all using delayed expansion:
"test.bat":

Code: Select all

set "line=a b c"
call that.bat "line"

"that.bat":

Code: Select all

@echo off
setlocal enableExtensions enableDelayedExpansion
echo(!%~1!
endlocal

Besdide this there's a flaw; the following code won't do what you might expect:

Code: Select all

echo ... 9>> "file"

Instead it redirects the scho output of outputstream 9 to the "file".

You better should redirect before the command, and if you are using a block only redirect at block definition:

Code: Select all

>> "file" echo ... 9
>> "file" (
   echo a
   echo b
)

Ad 3)
I don't get the point (probably because to me your notation seems to be unusual):
You should create a standalone example, where one could see the issue!

Ad 4)
What do you mean with "exploded".
Which result do you want to get, how do you try to achieve this, and what's the exact issue?

Ad 6)
I can't see a "recursive file inclusion" near line 179, and no "recursive static variable resolution" near line 285.
Probably i'm misunderstanding you, but knowing this doesn't help.

If you want to transform a recursive algorithm to an iterative one (which possibly could be faster - which isn't guaranteed), then you should create a minimal example, so wee could see and understand the algorithm you want to achieve (without trying to understand huge portions of that batch file).


Sidenotes:
Don't get me wrong, i really want to help!
But i don't want to read (huge parts) and get familiar with batch files of that size.
Let me give you an analogous example.
If i would have created a 1k lines batch file that somewhere includes my suggestion for solving issue 1 (see my above post), what would you think?
Would you try to understand that file even if i would have spread the solution in small parts across the whole batch, so the solution would be hard to find/understand?


penpen