READ/WRITE/DELETE/UPDATE records in sequential files

[miskox]

Hi all,

does anyone have a solution to UPDATE or DELETE records in a [line] sequential files (.txt files)?

Probably the only solution in DOS is to READ all records from a file and write a new file and then replace the original file with this newly written file. And this might take a very long time if there are many records just to delete/update one record.

Of course records that should be deleted are not written to the new file and with records which change (update) changes are written to the new file).

I really miss a function to do something like WRITE/UPDATE (which would replace the last read record from a file).

Saso

[aGerman]

Probably the only solution in DOS is to READ all records from a file and write a new file and then replace the original file with this newly written file.

This is the way like every other program does it as well (apart of some special applications like databases that manages their I/O buffers by themselves). If you own any recovery software you will see all the artefacts of old file versions (of course the memory is already released and will be overwritten as soon as it will be needed). The reason for this behavior is actually very simple. Every time you alter the content of a file you can't make sure the file and its meta data has exactly the same size as before. That's why you don't know if the file data fits into the memory space that was previously reserved for the file.
You're right that it will waste a lot of time using Batch but this is a question of the performance of Batch scripting in general.

Regards
aGerman

[Aacini]

A text file is a set of bytes with CR+LF pairs placed at fixed positions. For example, this data file:

Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday

Have 6 characters then a CR+LF pair, followed by another 6 characters (in the second line) and another CR+LF pair, etc. If the record update or delete you want may be done preserving the positions of the CR+LF pairs, then this process may be achieved with the aid of an auxiliary program. For example, a record may be "deleted" filling it with spaces, and a record may be modified as long as its original length is preserved. More complicated managements are also possible; for example, you may combine a deleted record with the previous one overwritting its CR+LF, or split a long record in two or more smallers records inserting CR+LF's, etc. The key for this to work is to preserve the records that are not modified in their same positions in the file. Of course, all these problems would be completely avoided with fixed length records...

Some time ago I wrote SetFilePointer auxiliary program that allows that previously described management be done; you just need to set the file pointer at the appropriate position and modify the bytes there. For example:

Code: Select all

(
   SetFilePointer 1 46
   echo Line 6
) >> Days.txt

Result:

Sunday
Monday
Tuesday
Wednesday
Thursday
Line 6
Saturday

Further details at this post. Please, note that SetFilePointer program described there is a 16-bits .com version that does not run in 64-bits Windows. I postponed the rewritting of this program, but if this method will solve your problem I could have the new version in 3 or 4 days...

Antonio

[miskox]

@Aacini:

this might be nice solution (quick). I will check this as soon as possible (I have a 32-bit XP PRO so 16-bit programs will work ok). I am only not sure if I will be able to copy/paste the code and generate the .com file.

I am also thinking of adding a special field in my .txt file to include a filepointer (fixed length of course) in the record itself so I know with every record I read what is its location in the file. I have variable length file records.

FILEPOINTER;NAME;SURNAME;SERIAL_NUMBER;ETC;ETC;ETC

FILEPOINTER would be a fixed length, for example PIC 9(9) (total file size is currently 2+ Mega bytes -so I need at least 7 characters for the filepointer).

As it is right now I might only need the UPDATE (REWRITE) option when serial number changes - which is always the same length).

Thanks for now.

Saso

[miskox]

@Aacini:

I can't make a .com file. Please provide a hex dump.

Thanks.
Saso

[Aacini]

What exactly is the problem? Run just this part:

Code: Select all

:CreateSetFilePointer
setlocal DisableDelayedExpansion
set SetFilePointer=2ÿ³‹û‹ó2íŠMÿãE° üó®t^>Š]ÿ€û0r6€û9w1€ë0Sã+€= u^&ó®t"Of3Àf3ɱ+€é!f3ÛGŠ]ÿ€ûEt+€ûeu(uÿë%%ëyGf3ÛŠ]ÿ€û0rs€û9wnf÷á€ë0f÷Ûf+ÃëáëØëäf‹È2Àf#ÉuTþÀþÀ€<EtK€<etFFÆDÿB‹Ñ±1€é!fÓé[´BÍ!r­€|ÿBu§±1€é!fÓâ‹Ðf÷Úf‹Ê±1€é!fÓé2ÀþÀþÀ´BÍ!2À´LÍ!ë¢ë½
setlocal EnableDelayedExpansion
echo !SetFilePointer!> SetFilePointer.com
exit /B

I just tested previous code in my Windows 8.1 computer and it correctly created the 219 bytes .com file. I compared it vs. the original SetFilePointer.com file, and it is OK!

Antonio

[miskox]

If I run your demo program (days) the .com uses 100% cpu and does not stop.

md5 of my .com is 0523AB79F1F3A9818E57FB2A1B19D062.

There might be problems with the code page or something so the .com is not correct.

Thanks.
Saso

[Aacini]

This is an hex dump of the 219 bytes of SetFilePointer.com file:

Code: Select all

32ffb3818bfb8bf332ed8a4dffe345b020fcf3ae743e8a5dff80fb30723680fb39773180eb3053e3
2b803d207526f3ae74224f6633c06633c9b12b80e9216633db478a5dff80fb45742b80fb6575288d
75ffeb25eb79476633db8a5dff80fb30727380fb39776e66f7e180eb3066f7db662bc3ebe1ebd8eb
e4668bc832c06623c97554fec0fec0803c45744b803c65744646c644ff428bd1b13180e92166d3e9
5bb442cd2172ad807cff4275a7b13180e92166d3e28bd066f7da668bcab13180e92166d3e932c0fe
c0fec0b442cd2132c0b44ccd21eba2ebbd0d0a

Please, report what happened when/if you discover it...

Antonio

[miskox]

Now it is working.

New MD5 hash is EDE16B37603C1D79074F34809ECF5EBA.

There are problems with copying/pasting strange characters.

My (non working) dump is:

Code: Select all

327933818B758BF332ED8A4D796145B020FCF3AE743E8A5D798075307236807539773180EB3053612B803D207526F3AE
74224F6633416633C9B12B80E921663355478A5D79807545742B80756575283F7579EB25EB79476633558A5D79807530
7273807539776E66F7E180EB3066F755662B41EBE1EB4FEBE4668B4532416623C975543F413F41803C45744B803C6574
4646414479428B4EB13180E92166D3E95BB442CD2172AD807C794275A7B13180E92166D3E28B3F66F7DA668B45B13180
E92166D3E932413F413F41B442CD213241B44CCD21EB63EB310D0A

First test shows that it is working.

I will try and make some READ/WRITE/UPDATE tests on my real data.

Thanks again.
Saso

[Aacini]

The idea of include the filepointer in the records themselves is very good, but it have not any advantage that this field be of fixed length. The method below would be a general update process over the file:

Code: Select all

(for /F "tokens=1-4* delims=;" %%a in ('findstr "KeyToFindRecordsToUpdate" theFile.txt') do (
   set /A "SERIAL_NUMBER=%%d+1"   // for example
   SetFilePointer 1 %%a
   echo %%a;%%b;%%c;!SERIAL_NUMBER!;%%e
)) >> theFile.txt

However, I remember that the use of the same file in both the FOR command and the >> redirection may cause some problems and I have not my old Win-XP computer right now. You should do a test; if you get unexpected results, you must divide previous process in two parts this way:

Code: Select all

rem Get records to update
findstr "KeyToFindRecordsToUpdate" theFile.txt > recordsToUpdate.txt

rem Update them
(for /F "tokens=1-4* delims=;" %%a in (recordsToUpdate.txt) do (
   set /A "SERIAL_NUMBER=%%d+1"   // for example
   SetFilePointer 1 %%a
   echo %%a;%%b;%%c;!SERIAL_NUMBER!;%%e
)) >> theFile.txt
del recordsToUpdate.txt

I suggest you to review this post for further details.

Antonio