An alternative command prompt (Dos9)

[aGerman]

I think I dont need to follow any cmd.exe behaviour, because the goal is not to get something fully compatible with cmd.exe (why no use cmd.exe for that purpose ? ).

Why do you expect somebody would use Dos9 instead of CMD then? (Please don't get me wrong.)

The goal is to provide a interpretor that is mostly compatible (I mean you can most batch files without modification) and to fix deficiences of cmd.exe.

The last part of your sentence is the point. But if you want anybody to use Dos9 you have to build a "mirror" of CMD first because Batch users would expect the same behavior. Then you can improve things and include extended functionality (like your realy useful SET /A:F). On the other hand you could also make Dos9 to behave like a console tool used in Batch code if necessary. Something like ...

Code: Select all

for /f "delims=" %%i in ('dos9 /printset /a:f 2/3+5') do set "float=%%i"

... or similar.

Regards
aGerman

[Darkbatcher]

Why do you expect somebody would use Dos9 instead of CMD then? (Please don't get me wrong.)

Ok What I wanted to say, is, well there's obviously behaviours that must be implemented (common behaviors), but there are also some others may not be implemented, for instance, strange behaviours that are used in hacks over cmd. I remeber of scripts I've seen that did things cmd wasn't supposed to do using these behavior. My position toward these is to let them freely hack cmd, but don't care if it actually works with Dos9. Of course, there are people that don't think so, for example, guys from the ReactOs project will probably tell you that "any known bug should be implemented" (and in this case "bug" is synonimous of "functionnality", brilliant ), that's another point of view...

The last part of your sentence is the point. But if you want anybody to use Dos9 you have to build a "mirror" of CMD first because Batch users would expect the same behavior. Then you can improve things and include extended functionality (like your realy useful SET /A:F). On the other hand you could also make Dos9 to behave like a console tool used in Batch code if necessary. Something like ...

Code: Select all

for /f "delims=" %%i in ('dos9 /printset /a:f 2/3+5') do set "float=%%i"

... or similar.

I agree, I have to make at least any behaviour you've mentionned (thank you very much BTW). I could also add a "/C" switch to handle that. But the syntax you gave is far too complicated, and too verbose, and it's sadly far from what most average batch users can achieve easily (at least on http://batch.xoo.it , but maybe french are dummies ). That's indeed a big drawbacks of cmd, it is not extendable, there's no plugin that can be used to add commands (for example, it would be usefull if you want to add GUI support).

Regards

[Squashman]

You think that FOR command is not achievable by the average batch user?
There have been plenty of 3rd party utilities written to extend BAT file functionality. Frank Westlake did a lot! He has a Utilities to add GUI support for selecting files and such.

[Liviu]

"CD X:", I did not even know about that. That's what's wonderfull with cmd, every new post teach me a brand new syntax

May I suggest you try "cd /?"

Type CD drive: to display the current directory in the specified drive.
Type CD without parameters to display the current drive and directory.

Use the /D switch to change current drive in addition to changing current directory for a drive.

With all due respect, but if you are to lay any claim to "cmd compatibility" you should first learn what 'cmd' itself does - and this is a case of an old, common, well documented behavior.

I not used wide characters because it is a mess on windows, but i can be fixed... to be curious, which unicode page are you using ? utf-8 or utf-16 ?

Sorry, don't know what you mean by "Unicode page". Windows itself uses UTF16 internally. What I showed was copy/pasted from a cmd prompt set to use Lucida Console (or another Unicode/TT - not raster) font.

What I wanted to say, is, well there's obviously behaviours that must be implemented (common behaviors), but there are also some others may not be implemented, for instance, strange behaviours that are used in hacks over cmd.

What you consider "strange" might be what others consider "common" - which is why it's important to define and delimit what Dos9 does (or doesn't). For an example, you called the '^!' parsing under enableDelayedExpansion "stupid" in an earlier post - and I won't argue that point However, that's been a fact of life with 'cmd' for a long time now, and there is code in many places specifically dealing with that anomaly. If Dos9 does it differently, all that existing code will break under Dos9 - and it's not just standalone batch files, but utility functions and macros as well.

I am not saying that Dos9 should follow every little idiosyncrasy of 'cmd' - that's your choice entirely, depending on your goals and target audience. But I am saying that it would add to Dos9's credibility if you listed a breakdown of missing/different features vs. just claiming it's "mostly compatible".

Liviu

[Darkbatcher]

You think that FOR command is not achievable by the average batch user?

Yes (it's a joke ) But for is a difficult part of batch, isn't it ?

There have been plenty of 3rd party utilities written to extend BAT file functionality. Frank Westlake did a lot! He has a Utilities to add GUI support for selecting files and such.

I know, I've written plenty of these stuffs. I also made utilities : one for adding GUi (yes with some limitation but gui), another to make graphical console (and much more), another to compute floating point expressions (there's link in a previous post).

And I experienced that plenty of users were unable to adapt it and just copy-pasted the code.

[Darkbatcher]

May I suggest you try "cd /?"

OH my god ! OH MY GOD ! ANOTHER BRAND NEW SYNTAX !

With all due respect, but if you are to lay any claim to "cmd compatibility" you should first learn what 'cmd' itself does - and this is a case of an old, common, well documented behavior.

Ok, haven't I the right to miss something ? :'( it's not really fair, isn't it ? I've already recognized that it was to be implemented. However, thank you for having coined this defieciencies.

I not used wide characters because it is a mess on windows, but i can be fixed... to be curious, which unicode page are you using ? utf-8 or utf-16 ?

Sorry, don't know what you mean by "Unicode page". Windows itself uses UTF16 internally. What I showed was copy/pasted from a cmd prompt set to use Lucida Console (or another Unicode/TT - not raster) font.

Ok, windows console support at least two unicode code page. The first is UTF-16 (1200) and the second is UTF-8. When you type something, your input gets translated either in utf-8 or in utf-16. This is important because if you use utf-16, NUL characters will be added (it's the case for 'm' which is an ASCII character), and, depending on the endianness, it may cause errors.

I am not saying that Dos9 should follow every little idiosyncrasy of 'cmd' - that's your choice entirely, depending on your goals and target audience. But I am saying that it would add to Dos9's credibility if you listed a breakdown of missing/different features vs. just claiming it's "mostly compatible".

[/quote]
Ok, I understood that I had to clone cmd. Well you're right.

Regardsi

[Squashman]

Yes (it's a joke ) But for is a difficult part of batch, isn't it ?

Once you understand what TOKENS are it is pretty much a no brainer from there.

[Squashman]

Seems like the best way to go is to make a Wrapper for CMD.exe.
Nice discussion about it over on SO. To many links to list here.
http://stackoverflow.com/questions/1977 ... eplacement

[Liviu]

Sorry, don't know what you mean by "Unicode page". Windows itself uses UTF16 internally. What I showed was copy/pasted from a cmd prompt set to use Lucida Console (or another Unicode/TT - not raster) font.

Ok, windows console support at least two unicode code page. The first is UTF-16 (1200) and the second is UTF-8. When you type something, your input gets translated either in utf-8 or in utf-16. This is important because if you use utf-16, NUL characters will be added (it's the case for 'm' which is an ASCII character), and, depending on the endianness, it may cause errors.

Still think you are misunderstanding, or maybe just misusing, the notion of "codepage". Windows is natively Unicode, and has been so since early NT times. It uses UTF16 encoding internally, specifically UTF16-LE in bytestreams where endianness matters, such as files on disk. As long as your program is built for Unicode (http://msdn.microsoft.com/en-us/library/dybsewaf.aspx) and/or explicitly uses the "W" versions of system functions (not "A" - e.g. ReadConsoleInputW not ReadConsoleInputA) you don't need to care about codepages at all (except in rare cases where you interact with _other_ programs which are not Unicode aware). Codepages are a fallback mechanism for the sake of legacy programs built for 8-bit character sets, where Windows attempts to do a remapping of codepoints back and forth - but that's unavoidably lossy, and locale dependent.

Also, codepage 1200 is an artifact for .NET (managed) apps, not really relevant to unmanaged console programs. And UTF-8, while it is supported as multi-byte codepage 65001, is not common in the console - in fact, as recently as XP, batch parsing was completely broken under codepage 65001.

Seems like the best way to go is to make a Wrapper for CMD.exe.

One can try and make a better (but different) cmd, or a compatible (but equally lousy) cmd. Though, not both in one

Liviu

[Darkbatcher]

Ok, windows console support at least two unicode code page. The first is UTF-16 (1200) and the second is UTF-8. When you type something, your input gets translated either in utf-8 or in utf-16. This is important because if you use utf-16, NUL characters will be added (it's the case for 'm' which is an ASCII character), and, depending on the endianness, it may cause errors.

Still think you are misunderstanding, or maybe just misusing, the notion of "codepage". Windows is natively Unicode, and has been so since early NT times. It uses UTF16 encoding internally, specifically UTF16-LE in bytestreams where endianness matters, such as files on disk. As long as your program is built for Unicode (http://msdn.microsoft.com/en-us/library/dybsewaf.aspx) and/or explicitly uses the "W" versions of system functions (not "A" - e.g. ReadConsoleInputW not ReadConsoleInputA) you don't need to care about codepages at all (except in rare cases where you interact with _other_ programs which are not Unicode aware). Codepages are a fallback mechanism for the sake of legacy programs built for 8-bit character sets, where Windows attempts to do a remapping of codepoints back and forth - but that's unavoidably lossy, and locale dependent.

Also, codepage 1200 is an artifact for .NET (managed) apps, not really relevant to unmanaged console programs. And UTF-8, while it is supported as multi-byte codepage 65001, is not common in the console - in fact, as recently as XP, batch parsing was completely broken under codepage 65001.

I understand what you say I said previously that I'm not using the unicode function (this is free, you can freely browse the code ). On reasonable OSes (I mean *NIX oses) there's no need for using wide characters (UTF-16), because it handles multibytes character (UTF-8, which have been surprisingly deprecated by Microsoft although it's probably the most common encode), in fact this is a far better approach, because any program that handles char* can handle utf-8 characters. Try to compile Dos9 on GNU/Linux or NetBsd and it will work without any encoding problem (Ok, that was the windows-sceptic moment ). I'm however right, if Dos9 get UTF-16 input, it actually reads garbage because there are NUL inserted between the ascii character. Of course, I'm not saying "Hey ! wide characters are shit, let's not use them", of course the problem should be addressed"

Seems like the best way to go is to make a Wrapper for CMD.exe.

One can try and make a better (but different) cmd, or a compatible (but equally lousy) cmd. Though, not both in one

Well, there is plenty of (free/Open-source) wrappers for cmd, like console2 or conEmu. But afaik, there's few cmd (compatible) replacements, i only know "Take Command" (aka 4NT) which is proprietary.

Regards

[penpen]

I'm not sure if i understand you right, you want to use the actual encoding of the a string as a byte list (char list)?
This may be no problem on one system, if all programs are using the same encoding, like it seems in *NUX (i haven't checked this).

But it also has multiple disadvantages:
- if you store resulting strings into file to disk, then these files have no defined content (as it bases on the systems encoding),
- you cannot avoid NUL characters which are not easy to type to a batch file,
- a system may support different character encodings at the same time (like windows) so a simple dir command may lead you into trouble, as the filenames are in UTF-16 LE: How do you want to display them?
- Think of support of other languages (on systems with multiple encodings) like for example the japanese language: Should they lookup the byte encoding of any of their characters to be able to do something like "dir *.j" (replace j with any japanese character) for example \U+98DF which is stored in batch on their pc as hex 90 48 (codepage 932) but the dir command gets the hex DF 98 as it probably uses UTF-16 LE; or if you retreive it with no unicode function then you may get ambiguous results: In the worst case you get a list of completely equal filenames consisting only of one replacement character. You can't fix that by using UTF-16 LE as batch file format, because then no actual batch file is able to work.
(see codepage 932: http://msdn.microsoft.com/en-us/goglobal/cc305152)
- ...

So you have to choose a more specific (== fixed over all systems) internal representation (if you want to use UTF-8, as it seems to be less work on *NIX, and you would have to deal with codepages in windows anyway), and you have to translate the input (files, pipes, copy-pastes, ...) to your internal representation, no matter whether you would like that or not.

penpen

[Darkbatcher]

I'm not sure if i understand you right, you want to use the actual encoding of the a string as a byte list (char list)?

Nope. What I said is it was sad that microsoft devs have choosed the behaviour that makes me forced of thinking about various encoding.
But well, that consist on replacing functions by their equivalent in <wchar.h> (yeah, I prefer platform independent functions). However, I read the documentation, and i't not clear enough, but, let's try ! Another option would be to read unicode strings and then convert them to utf-8 and then convert them back. It would be easier to implement but it would introduce overhead...

@+

[Ed Dyreen]

With "ver.EXE" one could fool a program in thinking the system runs another version of DOS, this was often necessary to make programs want to run where they would otherwise complain about "incorrect DOS version".

cmd.EXE behaves differently with every new version of windows. It would be nice if one could tell which version of cmd the script was written for so it would run this specific version and behave as expected.

Code: Select all

@version="8.0 (WinXP)"

If the annotation is left out, dos9 can be default. I believe this adds true value to Dos9 and would make people want to use it. Otherwise, it's just another scripting language.

My 2cent's

[Liviu]

Still think you are misunderstanding, or maybe just misusing, the notion of "codepage".

I understand what you say [...] I'm however right, if Dos9 get UTF-16 input, it actually reads garbage because there are NUL inserted between the ascii character.

That's not how it works in Windows. Dos9 is _not_ getting UTF16 input, because it's not _asking_ for it. Somewhere in the Dos9 code or libraries it uses, you end up calling the "A" flavor of the API for console input. So, when the user enters or pastes text at the prompt (which is fully Unicode and internally stored as UTF16 at that point), Windows sees that you are calling the narrow "A" (not wide "W") function - and takes that as a "please help, my code doesn't understand UTF16". Windows then does its best to help, and converts the UTF16 input to the codepage that Dos9 set for itself (or the default OEM codepage, in case it doesn't explicitly change it), and serves the converted string to the "A" function. Following that, your code receives the result as a "narrow" 8-bit-char-based string. There are no NULs or other "garbage" inserted, but the string may not be identical if the original contained characters which are not present in your particular codepage.

If Dos9 is to be Unicode aware in Windows, it must use the wide-char APIs, and make sure the external libraries do so, too. This doesn't mean that you must store and manage strings as UTF16 in your code. You can use any Unicode encoding internally, such as UTF-8 or UTF32. But the Windows API is UTF16, so whenever you call system functions you'd need to translate back and forth.

Liviu

[Darkbatcher]

That's not how it works in Windows. Dos9 is _not_ getting UTF16 input, because it's not _asking_ for it. Somewhere in the Dos9 code or libraries it uses, you end up calling the "A" flavor of the API for console input. So, when the user enters or pastes text at the prompt (which is fully Unicode and internally stored as UTF16 at that point), Windows sees that you are calling the narrow "A" (not wide "W") function - and takes that as a "please help, my code doesn't understand UTF16". Windows then does its best to help, and converts the UTF16 input to the codepage that Dos9 set for itself (or the default OEM codepage, in case it doesn't explicitly change it), and serves the converted string to the "A" function. Following that, your code receives the result as a "narrow" 8-bit-char-based string. There are no NULs or other "garbage" inserted, but the string may not be identical if the original contained characters which are not present in your particular codepage.

Ok ! so the use of unicode does not matter here, ansi input can still be used as long a you continue to type japanese on a japanese codepage (and that the codepage meets some conditions : to be ASCII compatible and not to have several codes that could map to ASCII characters).

But there's another problem, in msdn, the fgetws function documentation say that the file is read as multibyte characters. But I was wondering to what encoding was it really referring to, because if it is codepage-independent, reading batch would be qui inneficent since batch are written in plenty of codepages.

But i'm still not convinced by the translating of utf-16. Because usually, when I type "dir" in a folder where there are name like "trollé", the "é" is usually displayed as "ù" although there is an equivalent for "é" in the current codepage.

If Dos9 is to be Unicode aware in Windows, it must use the wide-char APIs, and make sure the external libraries do so, too. This doesn't mean that you must store and manage strings as UTF16 in your code. You can use any Unicode encoding internally, such as UTF-8 or UTF32. But the Windows API is UTF16, so whenever you call system functions you'd need to translate back and forth.

I think that will be what i'll do, because i've got quite a lot dependencies designed to handle utf-8 characters rather than utf-16.

Regards