DosTips.com

I discovered a couple problems when I want to use special and extended characters in parameters in a Batch file started with START command. This is the test.bat Batch file:
... that works perfectly when it is used this way:
... but I want to use special characters in the parameters, so I need to enclose they in quotes:
Ok. Now, I need to enclose the Batch file name in quotes because its path will contain spaces. Lets start with a test with no special characters first:
... that works OK. However, if I enclose the parameters in quotes (because they will contain special characters) the problems begins. This command:
... show this output in the started cmd.exe window:
After several tests, the only way I found to fix this point is include a CMD /K command after the START command AND insert an additional pair of quotes at the very beginning and end of the whole command, that is:
... that correctly works again.

However, if I include the special characters in the parameters:
... the problems returns...

After several tests I discovered that the only way to include special characters in the parameters in this way is escaping the special characters:
... that is not a convenient solution.

First question: is there any better way to include special characters in the parameters of a Batch file started this way?

Ok. If the problem is the characters included in the parameters, then just eliminate the parameters. I changed the test.bat file for this line:
... and executed it this way:
... that correctly show "First >" "Second |" in the started cmd.exe window.

===================================================================================

Ok. Lets pass now to the second problem. I want that the parameters include extended (beyond ASCII 127) characters:
This test correctly show the extended characters in the started cmd.exe window. However, this test was executed from the keyboard. Of course, I want to use the test.bat file from another Batch file, so I prepared it:
When I executed this second.bat file the started cmd.exe window show characters, but the characters are different:
I thought at first that I found the problem: both test.bat and second.bat were saved with the default Windows Notepad encoding: UTF-8. This point had caused me several problems in the past...

So I saved both files with ANSI encoding and checked that the characters in second.bat file be correct. Anyway, the characters shown ARE NOT THE SAME!
What happens here? How can I show the same extended characters of the calling Batch file in the called one?

Antonio

Hi Antonio.

1) There is a double parsing in place.
The caller parses the the quotes in the line as pairs. Only the characters within a pair are secured (if any ).
start "title" cmd /c ""script.bat" "arg1""

The syntax of the CMD /K (or /C) execution however requires the additional outer pair of quotes. The quoted tokens inside are still secured. (FWIW This proves that the cmd has a homebrew command line parsing.)
cmd /c ""script.bat" "arg1""

So, this conflicts. The solution is to only escape the outer quotes for the caller process. (The carets don't reach out to the second cmd process.)
start "title" cmd /c ^""script.bat" "arg1"^"


2) Actually the only way to process non-ASCII characters in a portable way in batch is to use UTF-8 (without BOM) everywhere. This "everywhere" means that you not only have to save the script UTF-8 encoded, but also that you have to tell the conhost process that it has to parse the whole I/O UTF-8 encoded. The latter only requires a
>NUL CHCP 65001
at the early beginning of the scripts.


caller.bat (UTF-8 no BOM)
test.bat (UTF-8 no BOM) Steffen

Wow! Both solutions worked as intended!

The escaped additional quotes are simple and works correctly.

To correctly manage the extended characters I must include a CHCP 65001 command AND save the Batch files with UTF-8 encoding. If one of these points is not completed, the method fails...

Thanks a lot!

Antonio

It's been always required that both the encoding of the script and the code page setting of the conhost process are the same.
Remember the days where we have been used to writing something like ... ... to draw boxes?
It took me quite a while to grasp the full extent of the fundamental wrongness of it.
Notepad and other editors didn't allow us to save the script in the default OEM code page. So we used the ANSI code page with characters having the same byte values as the box-drawing characters in the OEM code page. Thus, it was actually just a way to work around getting an OEM CP encoded script. However, this did only work for those people where the ANSI code page defaults to Windows-1252 and the OEM code page is at least a single-byte encoding (e.g. Japanese with CP 932 whould have had bad luck entirely). Writing the box drawing characters directly in the script and change the code page of conhost to CP 1252 is not possible just because those characters are not part of CP 1252. Conclusion is that the whole single byte code page stuff is crap and makes it a mess.
We have been also almost unable to do the right things though. At least not in a simple way. We should have taken a Unicode compliant encoding from the beginning. But typically editors on Windows prepended a BOM to a UTF-8 file that the command processor is still unable to handle/ignore.
However, at long last it became much easier and far more plausible now by just writing what we want to get printed ... ... and save the code UTF-8 encoded (without BOM). The emoji at the end is not a mistake. On recent Windows versions (>= Win10) we have a decent Unicode support in both console and terminal. We just need to jump on the bandwagon and make use of it Of course, you can even pass emojis to another script, as long as you align the encoding to UTF-8 throughout the whole chain: Steffen

damn I was expecting to see the emoji in the cmd window, but it didn't appear...

I used the same font as the editor, notepad, consolas, you can see them

its normal?

Missed part of this thread... I don't see it either:
Saso

The console/terminal team is permanently about to improve Unicode support. I defintely don't expect it to work on any version older than Win 10. But I actually thought that MS still backports new features to Win 10. I tested on Win 11 and don't have even access to a Win 10 machine anymore.

Console
Windows Terminal
Might also depend on the abilities of the font. "Cascadia Code" in my case.

Steffen

It's definitely because I'm on Windows 10. Anyway, thanks

I am on Win10 too.

But I am not moving to Win 11 just to see this smile.

Saso

For sure not because of a single smiley
It may still take a veeeery long time before we get close to the theoretical limit of 1114112 assigned Unicode code points. The majority is still not assigned. However, this number is 4352 times as many as conceivable in a single-byte encoding. Just to give a little indication about the difference it makes. Besides of all kinds of scripts and emojis, it contains mathematical symbols, shapes, dingbats etc.
The biggest advantage is that we don't have to rely on the right local settings. You can have Latin, Cyrillic, Greek, Chinese along with box drawings, blocks, ... all at the same time. So, ultimately we get what is state of the art on web sites for quite a long time (and the reason why we all see the same things here in the forum regardless where we are from). HTML defaults to UTF-8 if not explicitly stated otherwise. And now MS is working on making UTF-8 the standard for user interfaces (as it is a quasi standard on Unix-like operating systems already).

Steffen

I completely agree with you Steffen that this is going the right direction. But to have a .bat written for Windows 11 and majority of the users of this .bat are still on Windows 10 delays upgrade.

Thank you.
Saso