case study of the 'for %~sk' bug in win7

[Liviu]

Background: there was a longstanding xp bug where %~sk could return a wrong/invalid SFN in a for loop. Earliest reference I could locate is foxidrive's usenet post at https://groups.google.com/group/alt.msdos.batch.nt/msg/254df0764b2bbae8?hl=en. As my luck usually goes, I had to rediscover the bug on my own years later. In the ensuing discussion, it turned out that the original xp bug had been fixed in vista and later. However, a related (but likely different) bug still exists in Win7. Below is a somewhat simpler test case than the one I posted at http://groups.google.com/group/alt.msdos.batch.nt/msg/85379bd32ede38b6?hl=en though it still requires at least one character outside the current codepage to trigger it.

Assuming the following directory structure exists under the current directory, and for-sk-w7.cmd is...

Code: Select all

@for /d %%d in (*) do dir /x "%%~d"
for /r %%k in (*) do @if not exist "%%~sk" echo "%%~sk" ---???--- "%%~fk"

...the output under Win7 x64 is the following. EDIT: marked the files with an incorrect %~sk with ***.

Code: Select all

C:\tmp>for-sk-w7

C:\tmp>dir /x "19                €"

02/09/2013  11:39 PM                 0 12345_~1.1   12345=7.1
02/09/2013  11:39 PM                 0 1234_6~1.12  1234=6.12
02/09/2013  11:39 PM                 0 123_5~1.123  123=5.123
02/09/2013  11:39 PM                 0 12_4~1.123   12=4.1234

C:\tmp>dir /x "20                 €"

02/09/2013  11:39 PM                 0 12345_~1.1   12345=7.1    ***
02/09/2013  11:39 PM                 0 1234_6~1.12  1234=6.12
02/09/2013  11:39 PM                 0 123_5~1.123  123=5.123
02/09/2013  11:39 PM                 0 12_4~1.123   12=4.1234    ***

C:\tmp>dir /x "21                  €"

02/09/2013  11:39 PM                 0 12345_~1.1   12345=7.1
02/09/2013  11:39 PM                 0 1234_6~1.12  1234=6.12    ***
02/09/2013  11:39 PM                 0 123_5~1.123  123=5.123    ***
02/09/2013  11:39 PM                 0 12_4~1.123   12=4.1234

C:\tmp>dir /x "22                   €"

02/09/2013  11:39 PM                 0 12345_~1.1   12345=7.1
02/09/2013  11:39 PM                 0 1234_6~1.12  1234=6.12
02/09/2013  11:39 PM                 0 123_5~1.123  123=5.123
02/09/2013  11:39 PM                 0 12_4~1.123   12=4.1234

C:\tmp>for /R %k in (*) do @if not exist "%~sk" echo "%~sk" ---???--- "%~fk"

"C:\tmp\202075~1\12345_~1.1 \12345=7.1" ---???--- "C:\tmp\20                 €\12345=7.1"
"C:\tmp\202075~1\12_4~1.123 \12=4.1234" ---???--- "C:\tmp\20                 €\12=4.1234"
"C:\tmp\21716B~1\1234_6~1.12 \1234=6.12" ---???--- "C:\tmp\21                  €\1234=6.12"
"C:\tmp\21716B~1\123_5~1.123 \123=5.123" ---???--- "C:\tmp\21                  €\123=5.123"

C:\tmp>

What displays as an extra space in the directory part of the failing pathnames above is actually a U+2000 "en quad" space. Don't see any pattern, nor have much of a clue as to what kind of bug could possibly cause it. Just thought I'd mention it here for the cmd fun collectors

Liviu

[foxidrive]

Another thread a couple of years earlier. https://groups.google.com/d/topic/alt.m ... discussion

I found that using command.com helped but at the expense of a different issue.

[Liviu]

Another thread a couple of years earlier. https://groups.google.com/d/topic/alt.m ... discussion

Interesting, thanks for the pointer. Another link in that thread http://groups.google.com/group/alt.msdos.batch.nt/msg/32a14ae26b10e5cf dates the first XP bug report back to 2002, with a hint that it existed since the original NT.

Off the stated (Win7) topic now, the XP %~sk bug was arguably different, and definitely easier to hit, than the one still left in Win7. What they have in common is that among the necessary trigger conditions are a LFN directory name with a SFN strictly shorter, and a LFN file name with a SFN strictly longer. However, the additional dependency on "unicode" characters in the path applies to the Win7 bug, only. That said, the bug is not new to Win7 and existed in XP all along, just that it was obscured by the old non-Unicode-dependent bug. For the record...

Code: Select all

@dir /s /x
@echo. & for /d %%d in (*) do @for %%k in ("%%d\*") do @if not exist "%%~sk" echo "%%~sk" ---???--- "%%~fk"
@echo. & for /r %%k in (*) do @if not exist "%%~sk" echo "%%~sk" ---???--- "%%~fk"

...gives under XP Sp3 -- EDIT: marked the files with an incorrect %~sk with ***.

Code: Select all

C:\tmp>for-sk-xp

 Directory of C:\tmp\1 3     9

05/29/2010  02:51 PM                 0 120F6A~1.ABC 12.abcd
05/29/2010  02:51 PM                 0 123~1.ABC    123.abcd       ***
05/29/2010  02:51 PM                 0 1234~1.ABC   1234.abcd      ***
05/29/2010  02:51 PM                 0 12345~1.ABC  12345.abcd     ***
05/29/2010  02:51 PM                 0 123456~1.ABC 123456.abcd    ***
05/29/2010  02:51 PM                 0 123456~2.ABC 1234567.abcd
05/29/2010  02:51 PM                 0 123456~3.ABC 12345678.abcd

 Directory of C:\tmp\1 3    8

05/29/2010  02:51 PM                 0 120F6A~1.ABC 12.abcd
05/29/2010  02:51 PM                 0 123~1.ABC    123.abcd       ***
05/29/2010  02:51 PM                 0 1234~1.ABC   1234.abcd      ***
05/29/2010  02:51 PM                 0 12345~1.ABC  12345.abcd     ***
05/29/2010  02:51 PM                 0 123456~1.ABC 123456.abcd    ***
05/29/2010  02:51 PM                 0 123456~2.ABC 1234567.abcd
05/29/2010  02:51 PM                 0 123456~3.ABC 12345678.abcd

 Directory of C:\tmp\1 3   7

05/29/2010  02:51 PM                 0 120F6A~1.ABC 12.abcd
05/29/2010  02:51 PM                 0 123~1.ABC    123.abcd       ***
05/29/2010  02:51 PM                 0 1234~1.ABC   1234.abcd      ***
05/29/2010  02:51 PM                 0 12345~1.ABC  12345.abcd     ***
05/29/2010  02:51 PM                 0 123456~1.ABC 123456.abcd    ***
05/29/2010  02:51 PM                 0 123456~2.ABC 1234567.abcd
05/29/2010  02:51 PM                 0 123456~3.ABC 12345678.abcd

 Directory of C:\tmp\1 3  6

05/29/2010  02:51 PM                 0 120F6A~1.ABC 12.abcd
05/29/2010  02:51 PM                 0 123~1.ABC    123.abcd
05/29/2010  02:51 PM                 0 1234~1.ABC   1234.abcd
05/29/2010  02:51 PM                 0 12345~1.ABC  12345.abcd
05/29/2010  02:51 PM                 0 123456~1.ABC 123456.abcd
05/29/2010  02:51 PM                 0 123456~2.ABC 1234567.abcd
05/29/2010  02:51 PM                 0 123456~3.ABC 12345678.abcd

 Directory of C:\tmp\19                €

02/09/2013  11:39 PM                 0 12345_~1.1   12345=7.1      ***
02/09/2013  11:39 PM                 0 1234_6~1.12  1234=6.12      ***
02/09/2013  11:39 PM                 0 123_5~1.123  123=5.123      ***
02/09/2013  11:39 PM                 0 12_4~1.123   12=4.1234      ***

 Directory of C:\tmp\20                 €

02/09/2013  11:39 PM                 0 12345_~1.1   12345=7.1      ***
02/09/2013  11:39 PM                 0 1234_6~1.12  1234=6.12      ***
02/09/2013  11:39 PM                 0 123_5~1.123  123=5.123      ***
02/09/2013  11:39 PM                 0 12_4~1.123   12=4.1234      ***

 Directory of C:\tmp\21                  €

02/09/2013  11:39 PM                 0 12345_~1.1   12345=7.1      ***
02/09/2013  11:39 PM                 0 1234_6~1.12  1234=6.12      ***
02/09/2013  11:39 PM                 0 123_5~1.123  123=5.123      ***
02/09/2013  11:39 PM                 0 12_4~1.123   12=4.1234      ***

 Directory of C:\tmp\22                   €

02/09/2013  11:39 PM                 0 12345_~1.1   12345=7.1      ***
02/09/2013  11:39 PM                 0 1234_6~1.12  1234=6.12      ***
02/09/2013  11:39 PM                 0 123_5~1.123  123=5.123      ***
02/09/2013  11:39 PM                 0 12_4~1.123   12=4.1234      ***

"C:\tmp\139~1\123~1.ABCbcd" ---???--- "C:\tmp\1 3     9\123.abcd"
"C:\tmp\139~1\1234~1.ABCbcd" ---???--- "C:\tmp\1 3     9\1234.abcd"
"C:\tmp\139~1\12345~1.ABCbcd" ---???--- "C:\tmp\1 3     9\12345.abcd"
"C:\tmp\139~1\123456~1.ABCbcd" ---???--- "C:\tmp\1 3     9\123456.abcd"
"C:\tmp\138~1\123~1.ABCcd" ---???--- "C:\tmp\1 3    8\123.abcd"
"C:\tmp\138~1\1234~1.ABCcd" ---???--- "C:\tmp\1 3    8\1234.abcd"
"C:\tmp\138~1\12345~1.ABCcd" ---???--- "C:\tmp\1 3    8\12345.abcd"
"C:\tmp\138~1\123456~1.ABCcd" ---???--- "C:\tmp\1 3    8\123456.abcd"
"C:\tmp\137~1\123~1.ABCd" ---???--- "C:\tmp\1 3   7\123.abcd"
"C:\tmp\137~1\1234~1.ABCd" ---???--- "C:\tmp\1 3   7\1234.abcd"
"C:\tmp\137~1\12345~1.ABCd" ---???--- "C:\tmp\1 3   7\12345.abcd"
"C:\tmp\137~1\123456~1.ABCd" ---???--- "C:\tmp\1 3   7\123456.abcd"
"C:\tmp\19898B~1\12345_~1.1\12345=7.1" ---???--- "C:\tmp\19                €\12345=7.1"
"C:\tmp\19898B~1\1234_6~1.121234=6.12" ---???--- "C:\tmp\19                €\1234=6.12"
"C:\tmp\19898B~1\123_5~1.123123=5.123" ---???--- "C:\tmp\19                €\123=5.123"
"C:\tmp\19898B~1\12_4~1.123\12=4.1234" ---???--- "C:\tmp\19                €\12=4.1234"
"C:\tmp\204C83~1\12345_~1.1€\12345=7.1" ---???--- "C:\tmp\20                 €\12345=7.1"
"C:\tmp\204C83~1\1234_6~1.12\1234=6.12" ---???--- "C:\tmp\20                 €\1234=6.12"
"C:\tmp\204C83~1\123_5~1.123\123=5.123" ---???--- "C:\tmp\20                 €\123=5.123"
"C:\tmp\204C83~1\12_4~1.123€\12=4.1234" ---???--- "C:\tmp\20                 €\12=4.1234"
"C:\tmp\21C58B~1\12345_~1.1 €\12345=7.1" ---???--- "C:\tmp\21                  €\12345=7.1"
"C:\tmp\21C58B~1\1234_6~1.12€\1234=6.12" ---???--- "C:\tmp\21                  €\1234=6.12"
"C:\tmp\21C58B~1\123_5~1.123€\123=5.123" ---???--- "C:\tmp\21                  €\123=5.123"
"C:\tmp\21C58B~1\12_4~1.123 €\12=4.1234" ---???--- "C:\tmp\21                  €\12=4.1234"
"C:\tmp\2284D4~1\12345_~1.1  €\12345=7.1" ---???--- "C:\tmp\22                   €\12345=7.1"
"C:\tmp\2284D4~1\1234_6~1.12 €\1234=6.12" ---???--- "C:\tmp\22                   €\1234=6.12"
"C:\tmp\2284D4~1\123_5~1.123 €\123=5.123" ---???--- "C:\tmp\22                   €\123=5.123"
"C:\tmp\2284D4~1\12_4~1.123  €\12=4.1234" ---???--- "C:\tmp\22                   €\12=4.1234"

"C:\tmp\139~1\123~1.ABCbcd" ---???--- "C:\tmp\1 3     9\123.abcd"
"C:\tmp\139~1\1234~1.ABCbcd" ---???--- "C:\tmp\1 3     9\1234.abcd"
"C:\tmp\139~1\12345~1.ABCbcd" ---???--- "C:\tmp\1 3     9\12345.abcd"
"C:\tmp\139~1\123456~1.ABCbcd" ---???--- "C:\tmp\1 3     9\123456.abcd"
"C:\tmp\138~1\123~1.ABCcd" ---???--- "C:\tmp\1 3    8\123.abcd"
"C:\tmp\138~1\1234~1.ABCcd" ---???--- "C:\tmp\1 3    8\1234.abcd"
"C:\tmp\138~1\12345~1.ABCcd" ---???--- "C:\tmp\1 3    8\12345.abcd"
"C:\tmp\138~1\123456~1.ABCcd" ---???--- "C:\tmp\1 3    8\123456.abcd"
"C:\tmp\137~1\123~1.ABCd" ---???--- "C:\tmp\1 3   7\123.abcd"
"C:\tmp\137~1\1234~1.ABCd" ---???--- "C:\tmp\1 3   7\1234.abcd"
"C:\tmp\137~1\12345~1.ABCd" ---???--- "C:\tmp\1 3   7\12345.abcd"
"C:\tmp\137~1\123456~1.ABCd" ---???--- "C:\tmp\1 3   7\123456.abcd"
"C:\tmp\19898B~1\12345_~1.1\12345=7.1" ---???--- "C:\tmp\19                €\12345=7.1"
"C:\tmp\19898B~1\1234_6~1.121234=6.12" ---???--- "C:\tmp\19                €\1234=6.12"
"C:\tmp\19898B~1\123_5~1.123123=5.123" ---???--- "C:\tmp\19                €\123=5.123"
"C:\tmp\19898B~1\12_4~1.123\12=4.1234" ---???--- "C:\tmp\19                €\12=4.1234"
"C:\tmp\204C83~1\12345_~1.1€\12345=7.1" ---???--- "C:\tmp\20                 €\12345=7.1"
"C:\tmp\204C83~1\1234_6~1.12\1234=6.12" ---???--- "C:\tmp\20                 €\1234=6.12"
"C:\tmp\204C83~1\123_5~1.123\123=5.123" ---???--- "C:\tmp\20                 €\123=5.123"
"C:\tmp\204C83~1\12_4~1.123€\12=4.1234" ---???--- "C:\tmp\20                 €\12=4.1234"
"C:\tmp\21C58B~1\12345_~1.1 €\12345=7.1" ---???--- "C:\tmp\21                  €\12345=7.1"
"C:\tmp\21C58B~1\1234_6~1.12€\1234=6.12" ---???--- "C:\tmp\21                  €\1234=6.12"
"C:\tmp\21C58B~1\123_5~1.123€\123=5.123" ---???--- "C:\tmp\21                  €\123=5.123"
"C:\tmp\21C58B~1\12_4~1.123 €\12=4.1234" ---???--- "C:\tmp\21                  €\12=4.1234"
"C:\tmp\2284D4~1\12345_~1.1  €\12345=7.1" ---???--- "C:\tmp\22                   €\12345=7.1"
"C:\tmp\2284D4~1\1234_6~1.12 €\1234=6.12" ---???--- "C:\tmp\22                   €\1234=6.12"
"C:\tmp\2284D4~1\123_5~1.123 €\123=5.123" ---???--- "C:\tmp\22                   €\123=5.123"
"C:\tmp\2284D4~1\12_4~1.123  €\12=4.1234" ---???--- "C:\tmp\22                   €\12=4.1234"

C:\tmp>

Liviu

[foxidrive]

Interesting to read there are differing bags.

It would help if you highlight the lines that error - it's difficult to see what we are looking at on a simple inspection.

[Liviu]

It would help if you highlight the lines that error - it's difficult to see what we are looking at on a simple inspection.

Right. Done, with *** since colors don't seem to work inside code tags.

While the legacy XP bug has a discernable pattern, the Win7 one looks quite odd. Maybe someone with more imagination (and spare time ) can guess the hidden failure rule.

[Squashman]

quote tags will show color.

[Liviu]

Thanks, and right. But quote tags also coalesce consecutive spaces, and use a proportional font, instead. Guess one can't always have it both ways. Which should be a familiar line around here

[foxidrive]

Thanks, I see that these generate the wrong output lower down the page.

Code: Select all

05/29/2010  02:51 PM                 0 123~1.ABC    123.abcd       ***
05/29/2010  02:51 PM                 0 1234~1.ABC   1234.abcd      ***
05/29/2010  02:51 PM                 0 12345~1.ABC  12345.abcd     ***
05/29/2010  02:51 PM                 0 123456~1.ABC 123456.abcd    ***

Such as in here:

"C:\tmp\139~1\123~1.ABCbcd" ---???--- "C:\tmp\1 3 9\123.abcd"