findstr with regex to find characters with MSB bit set

[miskox]

Hello!

I am strugling with regex. Let's say I have a test.txt (one byte characters) file (50 bytes, NOTEPAD shows ANSI bottom right):

Code: Select all

testline1
somethingelse
Šmoretext
anotherline

(Š above has a value of 0x8A - it doesn't matter which character you insert if bit7 is set)
I want to find lines with characters that have MSB bit set (bit 7).

No luck:

Code: Select all

c:\findstr /R "[^\x00-\x7F]" test.txt

returns no lines. I need this line:

Code: Select all

Šmoretext

Code: Select all

C:\>findstr /R "[\x8a]" test.txt

returns:

Code: Select all

Őmoretext
anotherline

instead of

Code: Select all

Őmoretext

This would be a start for my task: ideally I would like to find all lines that have a character that is different from:

valid characters:
LF: 0x0A
CR: 0x0D
Č: 0xAC
Š: 0xE6
Ž: 0xA6
č: 0x9F
š: 0xE7
ž: 0xA7
and 0x20-0x7f

Any ideas?
Thanks.
Saso

(I really don't have much experiences with regex)

[aGerman]

NOTEPAD shows ANSI

My guess is that the default ANSI code page is Windows-1250 in your case.
However, ...

Č: 0xAC
Š: 0xE6
Ž: 0xA6
č: 0x9F
š: 0xE7
ž: 0xA7

--- are character codes for CP 852.
I really doubt that your test.txt file is CP 852 encoded though. Could you shed some light on that?

Besides of that, don't expect that FINDSTR supports RegEx in a usual way. It doesn't know anything about hex expressions like \xHH.

Steffen

[miskox]

@Steffen: you are correct about the CP1250 and CP852 - but it is not important for the test because bit 7 is set.

Looks like this is causing the problems:

Besides of that, don't expect that FINDSTR supports RegEx in a usual way. It doesn't know anything about hex expressions like \xHH.

Is there a solution for that?

Thanks.
Saso

[aGerman]

but it is not important for the test because bit 7 is set.

Oh, I've been under the impression that you wanted to exclude those characters since you have them in the list of "valid characters". Hmm...

Is there a solution for that?

Yes and no.
Yes - You may use 3rd party utilities like SED or GRAP. You could use an existing JScript hybrid like JREPL. And you could write your own code in another language like VBScript, JScript, or PowerShell.
No - I don't see any possibility to use FINDSTR for this task.

Steffen

[miskox]

Thanks Steffen!

Currently I am dumping my .txt file with your fildump - and I will proceed from there (it is ~33.000 lines, 4+MB in size). After that I will see what to do. The fact is that this might be a one time test only (or maybe twice a year) so I don't really need a quick solution. Next option is: I can write an .exe (now I use PureBASIC because my old DOS COBOL is not good anymore).

Thanks again.
Saso

[aGerman]

It's not too complicated.

Code: Select all

cmd /c jrepl.bat "^[\x09\x20-\x7F]*$" "" /N 1 /R 0 /XSEQ /F "test.txt"

You can add more valid characters right after \x7F. Use the char codes of Windows-1250 in this case. (FWIW I started with \x09 because I guess you don't want tab characters reported.)

Steffen

[miskox]

Thanks. Works like a charm! Found some invalid characters. Fixed. I added some more of course (for example ü is a valid character because we have some people with Müller as the last name for example).

Saso