CONVERTCP.exe - Convert text from one code page to another

[jfl]

how many bytes do you actually stop trying to figure out what encoding you got?

Currently I scan the whole file after loading it in memory, counting even NULs, odd NULs, non-ASCII bytes, invalid UTF-8 sequences. Then in the end, based on these counts, I select the most likely encoding among UTF-16, UTF-8, or ANSI.
A simple improvement could be to abort the scan when any counter passes a given limit. I'm pretty sure the impact on performance would be negligible.

[aGerman]

Since I implemented an own UTF-8 conversion algorithm I also have to validate char by char in this case because I have to shift in a replacement character if an invalid sequence is found. But everything else would conflict with the design of CONVERTCP. Usually I read only up to 1 MB at once to be able to read, convert, and write in threads. This has been a huge performance improvement for big files. Validating the whole chunk of data before actually converting it is what currently keeps me from supporting multithreading for DBCS code pages. Other than in UTF-8 and UTF-16 encoded text you would have to iterate over the entire chunk to figure out if it ends at a lead byte or if you got the whole character at the buffer boundary.

Steffen

[aGerman]

As lately discussed with Jean-François I find it a pretty good idea to print the output as UTF-16 whenever the app detects a text device (such as a console or a terminal). Thank you for this suggestion!
In these cases the CP_out argument will simply not be used even if you still have to pass it for syntax reasons. That way the possible characters are not restricted by the character set anymore. Limits only exist due to the characters the font supports or due to the ability of the terminal to output glyphs outside of the Basic Multilingual Plane.
I tested in the usual conhost window as well as in the Windows Terminal and in ConEmu. The Windows Terminal supports even the output of emojis.

Virustotal scans of version 7.5:
x86: https://www.virustotal.com/gui/file/1c1 ... /detection
x64: https://www.virustotal.com/gui/file/ca4 ... /detection

Steffen

[aGerman]

I can't believe to find myself doing this ...

As of version 8 CONVERTCP supports the codepage detection of the incoming stream. It's still rather guessing. So, you should not rely on it.

Jean-François made some tests with IMultiLanguage2::DetectInputCodepage and the results have been exciting enough to investigate how to improve it.
viewtopic.php?f=3&t=10088

If you want to try out, just use a question mark as first argument.

Code: Select all

convertcp ? 65001 /i "a.txt" /o "b.txt"

CONVERTCP will try to determine the encoding of a.txt for the conversion.


You can also hint an encoding. Use the question mark followed by a preferred codepage ID. If reasonable, CONVERTCP will use it rather than the guessed encoding. However, if the guessed encoding is more likely than the hint, CONVERTCP won't use your preference.

Code: Select all

convertcp ?437 65001 /i "a.txt" /o "b.txt"

In both cases CONVERTCP will fail (no conversion performed) if it is unable to guess an encoding.


Virustotal scans of version 8.0:
x86: https://www.virustotal.com/gui/file/cac ... 96d1c78c7a
x64: https://www.virustotal.com/gui/file/8a5 ... f7ee9a5385

Steffen

[findstr]

...
Virustotal scans of version 8.0:
x86: https://www.virustotal.com/gui/file/cac ... 96d1c78c7a
x64: https://www.virustotal.com/gui/file/8a5 ... f7ee9a5385

Steffen

Are those false positives, or the program may possibly be malware?

[aGerman]

What do you think?
Have a look at post #72 in this thread where I hopefully explained it all. viewtopic.php?p=57914#p57914

Steffen

[miskox]

Steffen, thank you. Yes.. you thought there will be no more updates...

I will give it a try but might take a week or two.

Thank you.

Saso

[aGerman]

Hi Saso,

It's years ago when I thought that But actually this update is something where I was quite sure that I will never implement it. Meanwhile I'm quite positive that codepage detection is something useful as long as it is better than just a gambler's game. Even though it will never be foolproof ¯\_(ツ)_/¯

However, even if this is something that you may never use, it would be nice to know how downward-compatible the tool still is. So, yeah, if you still have the opportunity to try on XP I'd be happy to get some feedback on that

Steffen

[aGerman]

Saso verified that the tool is still running on XP. Of course he also gave some hints that codepage detection is everything but perfect. It turns out that files with a reasonable amount of text work way better than files with only a few words. That's been expected. However, also files which don't consist of natural language text tend to fail. E.g. ANSI-encoded CSV data which contains an unusual amount of commas (or semicolons) and numbers, compared to natural language only. They influence the statistical byte distribution which the API functions expect to see. That's nothing I'm able to improve though.

He also reminded me that the tool still throws error messages that are rather misleading. Should be better now. Also UTF-7 recognition was somewhat broken and is fixed in v. 8.1.

EDIT: I made a mess with the error messages I'm updating the tool but leaving the version.

Virustotal scans of version 8.1:
x86: https://www.virustotal.com/gui/file/3fe ... b51ffab672
x64: https://www.virustotal.com/gui/file/94a ... a55f94e8f5

Steffen

[aGerman]

Version 8.2 is a minor update. It adds the opportunity to overwrite the original file with the converted content. Pass a single minus sign along with option /o to specify it.

The minus sign is only taken as wildcard for the same file name under the following assumptions:
- option /i is specified
- option /a is not specified

The functionality is the same as previously recommended, but is automated now:
- a temporary file is created in the same directory (original file name with an appended GUID string)
- if the conversion is successful, the original file will be replaced by the temporary file

Virustotal scans of version 8.2:
x86: https://www.virustotal.com/gui/file/09d ... 4dfd848878
x64: https://www.virustotal.com/gui/file/d4d ... 4e740fc1a9

Steffen

[miskox]

Steffen, you can't stop?

I mean - thanks.
Saso

[aGerman]

Version 8.3 improves the performance of UTF-8 identification. Further I found a way to shrink the binary size by ~20 KB.

Virustotal scans of version 8.3:
x86: https://www.virustotal.com/gui/file/4d1 ... d8fb1a3172
x64: https://www.virustotal.com/gui/file/e5a ... 8a46ec4c4a

Steffen

[aGerman]

Only one parenthesis at a wrong position in the source code broke the recognition of MIME names

Virustotal scans of version 8.4:
x86: https://www.virustotal.com/gui/file/470 ... 28561757c5
x64: https://www.virustotal.com/gui/file/b3c ... d7396c87db

Steffen

[shantanu97]

The below code is showing an error:Unable to create output file. The input folder contains a bunch of CSV files and the output folder name is (UTF8Files) where we are saving a bunch of CSV files in UTF-8 encoding. Remember my input folder locations and output folder locations are different.

Code: Select all

md "C:\Users\ShantanuGupta\Documents\Test\UTF8Files"
for %%i in (C:\Users\ShantanuGupta\Desktop\UTF8\Input\*.csv) do convertcp.exe ? 65001 /i "%%~i" /o "C:\Users\ShantanuGupta\Documents\Test\UTF8Files\%%~i"
pause

Can anyone help me why my code is failing.

[aGerman]

%%~i contains the whole path of the file. If you read the help message of FOR (run FOR /? in a cmd prompt) you'll find the modifiers of FOR variables. E.g. %%~nxi expands to the file name and file extension which is likely what you're looking for.

Steffen