Windows 7 has Black Holes - Data is lost with no retrieval.

[alan_b]

A Blackhole is where you create a file or folder in W7 and it disappears out of sight
with no way to access it or retrieve any information.
It has fallen through the Blackhole and travelled through a Wormhole to another place.

A BlackHole exists at "%USERPROFILE%\Application Data\"
Any File or Folder which is sent there will appear at
C:\Users\%USERNAME%\Appdata\Roaming
but DIR "%USERPROFILE%\Application Data\" actually looks at
"C:\Users\%USERNAME%\Application Data\" and cannot see anything
DIR cannot find it
DEL cannot delete it
IF EXIST shows it does NOT exist
CACLS bombs out with "access denied"
CD has only partial success - it fails if you include a wildcard in the path
RD seems to work even though DEL does not.

After testing all the relevant DOS commands I can think of,
I believe a BlackHole is Write Enabled and effectively Read Disabled
which explains why CD {specific path\folder} can work without reading,
but CD {specific path\wildcard-folder} fails because it is not allowed to look for a wild-card match.

To illustrate the problem I WISH TO attach Wormhole.gif which shows creation of 3 files on 3 paths, unfortunately attachments are not allowed.
Just seen the Img button and done frantic search, found image host :-
[img][img]http://a.imagehost.org/0235/Wormhole.gif[/img][/img]

All three are written to the destination C:\Users\Alan\Appdata\Roaming,
but only "%APPDATA%"\ALAN_20 avoids Junctions and has full read/write/modify access,
whilst ALAN_21 and ALAN_22 are doomed to write only, and cannot be typed or deleted.

You can try for yourself if you paste into a CMD.EXE DOS Window all of the following, including the blank line :-

Code: Select all

echo %TIME%
echo %TIME% >> "%APPDATA%"\ALAN_20
echo %TIME% >> "%USERPROFILE%\Application Data"\ALAN_21
echo %TIME% >> "C:\Documents and Settings\%username%\Application Data"\ALAN_22
echo %TIME%
TYPE "%APPDATA%"\ALAN_20
TYPE "%USERPROFILE%\Application Data"\ALAN_21
TYPE "C:\Documents and Settings\%username%\Application Data"\ALAN_22
TYPE "%APPDATA%"\ALAN_21
TYPE "%APPDATA%"\ALAN_22
DIR "%APPDATA%"\ALAN_* | FIND ":"
DIR "%USERPROFILE%\Application Data"\ALAN_* | FIND ":"
DIR "C:\Documents and Settings\%username%\Application Data"\ALAN_* | FIND ":"
DEL "%APPDATA%"\ALAN_20
DEL "%USERPROFILE%\Application Data"\ALAN_21
DEL "C:\Documents and Settings\%username%\Application Data"\ALAN_22

I actually pasted ran the above code 4 times in the same DOS window,
hence ALAN_21 and ALAN_22 include 3 previous timestamps and demonstrate delete prohibition.
When that finishes the DOS window remains available for you to poke around and investigate further.

Copy the following into a file WT_OOPS.BAT

Code: Select all

@echo off & setlocal enabledelayedexpansion & CLS
SET MODE=FILE
IF "%LOCALAPPDATA%"=="" SET LOCALAPPDATA=%USERPROFILE%\Local Settings\Application Data
IF "%USERPROFILE%"=="C:\Documents and Settings\Dad" SET USERNAME=Dad

SET SAFE="c:\ALAN\AbsentFrozen_Path_sub."
IF NOT EXIST %SAFE%\ MD %SAFE%

IF "%MODE%"=="SUB" (CALL :P MD) ELSE (CALL :P ECHO)
CALL :P EXIST
CALL :P CACLS
IF "%MODE%"=="SUB" (CALL :P CD) ELSE (CALL :P TYPE)
CALL :P DIR
IF "%MODE%"=="SUB" (CALL :P RD) ELSE (CALL :P DEL)
IF EXIST C:\Users\ CALL :P DIR
GOTO :EOF

:P
ECHO(
SET /A N=10
CALL :X %1 "%%%%USERPROFILE%%%%\
CALL :X %1 "C:\Documents and Settings\%%%%USERNAME%%%%\
IF EXIST C:\Users\ CALL :X %1 "C:\Users\%%%%USERNAME%%%%\
SET /A N=20
CALL :X %1 "%%%%APPDATA%%%%\
CALL :X %1 "%%%%USERPROFILE%%%%\Application Data\
CALL :X %1 "C:\Documents and Settings\%%%%USERNAME%%%%\Application Data\
SET /A N=30
CALL :X %1 "%%%%LOCALAPPDATA%%%%\
CALL :X %1 "%%%%USERPROFILE%%%%\Local Settings\Application Data\
CALL :X %1 "C:\Documents and Settings\%%%%USERNAME%%%%\Local Settings\Application Data\
SET /A N=40
CALL :X %1 "%%%%ALLUSERSPROFILE%%%%\
CALL :X %1 "C:\Documents and Settings\All Users\
IF EXIST C:\Users\ CALL :X %1 "C:\Users\All Users\
SET /A N=50
CALL :X %1 "%%%%ALLUSERSPROFILE%%%%\Application Data\
CALL :X %1 "C:\Documents and Settings\All Users\Application Data\
IF EXIST C:\Users\ CALL :X %1 "C:\Users\All Users\Application Data\
GOTO :EOF

:X
CALL set P=%2
ECHO X %1 %2ALAN_" %N%
IF "%1"=="MD" %1 %P%ALAN_%N%"
IF "%1"=="DIR" %1 %P%ALAN_*" /A | FIND ":"
IF "%1"=="CD" CD /D %SAFE% & %1 %P%ALAN_*" & CD & CD /D %SAFE% & %1 %P%ALAN_%N%" & CD
IF "%1"=="RD" CD .. & ATTRIB -R -A -S -H %P%ALAN_%N%" & %1 /S /Q %P%ALAN_%N%"
IF "%1"=="EXIST" IF EXIST %P%ALAN_%N%" (ECHO VALID  %P%ALAN_%N%") ELSE ECHO ABSENT %P%ALAN_%N%"
IF "%1"=="CACLS" %1 %P%ALAN_%N%" > NUL && (ECHO VALID  %P%ALAN_%N%") || (ECHO WRONG  %P%ALAN_%N%")
IF "%1"=="ECHO" %1 %TIME% >> %P%ALAN_%N%"
IF "%1"=="DEL" ATTRIB -R %P%ALAN_%N%" & %1 %P%ALAN_%N%"
IF "%1"=="TYPE" %1 %P%ALAN_%N%"
SET /A N+=1

P.S. Change the second line of code from "SET MODE=FILE" to "SET MODE=SUB" to observe similar effects using SUBfolders instead of Files.

Now you can perform the extensive tests I performed on 15 paths leading to 5 destinations in XP,
which merged into 4 destinations in W7.
Some Junctions appear to allow full access where others greatly restrict access.

Windows 7 has perhaps 44 different Blackholes.
This is the number of JUNCTIONS detected by Junction Link Magic under XP on my dual boot PC.
I do not know if that excludes symlinks and other forms of reparse points in W7.
I am not sure now if the Junctions that allowed read access were actually symlinks or something else,
I just know and am warning that there are black holes in W7 that gobble up data but do not allow it to escape.

There is an urban myth that M.$. use JUNCTIONS etc so VISTA and W7 can use software created for XP.
My evidence shows a great loss of functionality when JUNCTIONS are used for this purpose.

I believe M.$. "borrowed" the idea of Junctions from UNIX,
but failed to understand it and implemented a disastrous side effect,
i.e. deleting a Junction could also destroy everything it was aimed at.
After people complained about lost data M.$ chose not to fix the error,
but recommended a bodgy workaround, i.e.
"Use NTFS ACLs to protect junction points from inadvertent deletion" as per
http://support.microsoft.com/kb/205524/en-us
I guess it is an effect of the ACLS that I cannot read read through a Junction.

When I use CACLS to investigate the ACLS of various Junctions I am bewildered by the confusion.
For some Junctions CACLS reports "access denied";
For some there is amongst other things "Everyone:(DENY)(special access:)"
and others have "Everyone:(DENY)(special access:)" together with "Everyone:R"
I saw many other variations in permissions for various Junctions.
I wonder if all of these inconsistently protected Junctions have equal protection against deletion ! !

With such inconsistency it looks like Vista/W7 has a broken leg,
and tech support tried to mend it with a sticky plaster.

Some Junctions provide W7/XP backward compatibility,
and allow a CMD.EXE script to use XP style paths and fully access the relevant locations in W7
Many Junctions give almost zero access to the relevant locations in W7.
If Junctions break the functionality of XP batch command files,
I think it must be even worse for GUI type applications that were written for XP.

NB For my present needs I am rewriting some scripts to make maximum use of environmental variables.
Using %userprofile% is a step in the right direction,
but NOT if a JUNCTION then diverts "Application Data" in which case %appdata% is needed,
but not if .........

I never saw any warnings in any forums, and a Google search fails to find it on the Internet.

Am I the only one to see that "The Emperor has no Clothes" ! ! !

Alan

[orange_batch]

Interesting.

Are you just testing OS behaviour, or does this affect your scripting somehow?

It may be pointless or known to them, but have you thought of reporting it to Microsoft?

[alan_b]

This is not a casual test of O.S.'s, it is research into how W7 has broken my CMD.EXE and my script.

I am contending with an issue that is fatal to the installation of Comodo Internet Security,
and probably fatal to alternative security products,
and fatal to the operation of scripts.

Upgrading from an ancient Comodo to a recent version can fail if the old version is not first adequately removed.
Normally a simple ADD/REMOVE un-installation is adequate, but not always.
Sometimes a "Clean-up" script is needed to eliminate known "pockets of resistance"

I am not satisfied by the script posted on their forum because it ASSUMES that CMD.EXE will remove a file or registry key on request, and it ignores ACCESS DENIED messages arising from "file in use" and permissions issues, and the ignored remnants may still block a successful installation.

I once had to insert dozens of PAUSE breaks in the script before I could see what had Access denied, and then I found that a tree of 2000 registry keys were the problem, and Regedit could not delete them. Eventually I learnt how to "take ownership" and then I could delete them and then install the latest version.

I improved the script by adding a validate stage that after deleting "pockets of resistance" it would then visit them a second time and NOT delete but list and fully identify everything that remained and needed extreme manual efforts.

I tested my script by deliberately adding to the system some registry keys for which no entity had access,
and would only go on my command after I took ownership.
I also added files and folders and used CACLS to block all access.
I created files with the names of folders that the script would try to RD, and due to failure an upgrade would be blocked
I created folders with the names of files the script would try to DEL
I had CMD.EXE sit in the folder that had to be deleted so Windows would say the folder was in use.
I launched PING and piped its output into a file so Windows would say the file was in use.

I threw every obstacle I could think of, and my script perfectly detected and reported on the second stage all the things that had not been eliminated on the first stage.

Perfection on XP.

As a kindness to W7 users I checked that the script was good for W7 as well as XP.
That is when I found that W7 is not good for scripts.

The first stage of deleting APPEARED as effective on W7 as on XP,
but in fact some files FAILED to be deleted and there was no warning at all
The second stage was a failure
IF EXIST told the script that a designated file did not exist, even though it did.
Most CMD.EXE commands fail to follow the path of a Junction which allows only CREATE actions,
and instead they are diverted into dummy paths and then report they cannot see the file/folder they should read/delete/etc.

I only know of this problem because I deliberately created these un-deletable items to test my error reporting,
and instantly recognised a failure to report on some of the many items I expected to be found,
and then I found it was not the fault of my script which I was testing, but a fault in Windows 7.

I would be interested in learning whether Vista has the same problems as W7.

I believe what I have found is likely to have an adverse effect on any script that might suffer a Junction diversion,
hence I posted here to warn others and to receive any relevant information that may be offered.

I never thought of notifying Microsoft.

Is there any particular site/forum to which I should put this information,
or do you suggest I report this to Tech Support

Regards
Alan

[orange_batch]

I'm not sure. There must be some kind of particular bug report for Windows 7.