Desired number of spaces - optimizing the code?

[miskox]

Hi all!

I have a subprogram to make a variable with desired number of spaces:

Code: Select all

@echo off
echo %TIME%
call :makespaces 100
echo %TIME%
goto :EOF

:makespaces
set spaces=&rem
set par1=%1&rem
set /a cntspaces=0
:spacesback
set spaces= %spaces%
set /a cntspaces+=1
if %cntspaces% LSS %par1% goto :spacesback
goto :EOF

This subprogram runs fast enough if the file size (of .cmd) is relatively small. If I add some few thousands of lines it goes very slow.

Less than a second if this is very short program. I added 60000 records (does not matter if they are at the beginning of a file or at the end) execution time increases to 24 seconds.

Any ideas? (I prefer not to use delayed expansion and such commands).

Would it run faster if it would be a FOR /L loop (I cannot make it to work)?

Thanks.
Saso

[Aacini]

The subroutine should run faster if a FOR command is used, but this requires to use delayed expansion:

Code: Select all

:makespaces
set "spaces="
for /L %%i in (1,1,%1) do set "spaces= !spaces!"
goto :EOF

However, it would be even faster if you prepare in advance a variable with the maximum number of possible spaces and just take a part of it in the subroutine:

Code: Select all

:makespaces
call set "spaces=%%maxSpaces:~0,%1%%"
goto :EOF

This method eliminate the need of have a separate subroutine, so you may directly insert the previous line in the code instead of "call :makespaces N".

Antonio

[Squashman]

And of course using the dreaded temporary file.

Code: Select all

@echo off
<nul set /p ".=1 " >dummy.txt
for /l %%n in (1 1 10) do type dummy.txt >>dummy.txt
set /P blanks=<dummy.txt
del dummy.txt
set "blanks=%blanks:1=%"
echo %blanks:~0,1%ONE
echo %blanks:~0,10%TEN
pause

output

Code: Select all

 ONE
          TEN
Press any key to continue . . .

[Squashman]

Depending on what you are trying to do you could use the :FORMAT function. I added the END to show you that it is formatting the string with trailing spaces. And as you can see in the output you can also do leading spaces.

Code: Select all

@echo off
call :Format "[20]" Column1
echo %line%END
call :Format "[10][20]" Column1 Column2
echo %line%END
call :Format "[10][20][30]" Column1 Column2 Column3
echo %line%END
call :Format "[-20]" Column1
echo %line%END
call :Format "[-10][-20]" Column1 Column2
echo %line%END
call :Format "[-10][-20][-30]" Column1 Column2 Column3
echo %line%END
pause
GOTO :EOF

:Format fmt str1 str2 ... -- outputs columns of strings right or left aligned
::                        -- fmt [in] - format string specifying column width and alignment, i.e. "[-10][10][10]"
:$created 20060101 :$changed 20091130 :$categories Echo
:$source http://www.dostips.com
SETLOCAL
set "fmt=%~1"
set "line="
set "spac=                                                     "
set "i=1"
for /f "tokens=1,2 delims=[" %%a in ('"echo..%fmt:]=&echo..%"') do (
    set /a i+=1
    call call set "subst=%%%%~%%i%%%spac%%%%%~%%i%%"
    if %%b0 GEQ 0 (call set "subst=%%subst:~0,%%b%%"
    ) ELSE        (call set "subst=%%subst:~%%b%%")
    call set "const=%%a"
    call set "line=%%line%%%%const:~1%%%%subst%%"
)
endlocal&set "line=%line%"

GOTO :EOF

output

Code: Select all

Column1             END
Column1   Column2             END
Column1   Column2             Column3                       END
             Column1END
   Column1             Column2END
   Column1             Column2                       Column3END
Press any key to continue . . .

[aGerman]

(I prefer not to use delayed expansion and such commands).

That doesn't make sense to me.

Code: Select all

@echo off &setlocal
echo %time%
call :makespaces 100
echo %time%
pause
exit /b

:makespaces
setlocal EnableDelayedExpansion
set "spaces= "
for /l %%i in (1 1 12) do set "spaces=!spaces!!spaces!"
set "spaces=!spaces:~,%1!"
endlocal &set "spaces=%spaces%"

The code creates a sequence of 4096 spaces within only 12 iterations. Delayed Expansion will be disabled again after leaving the sub routine.
You could work with a GOTO loop as well but why making things more complicated and slower?

Regards
aGerman

[miskox]

Sorry for a short delay (have been out of the country for a few days on business).

I am doing a program to maintain an indexed sequential-based file (see viewtopic.php?p=34025 for more info).

When I delete/update a record that has a different length than the original record I must replace current record with spaces (in the same length). Though in practice records do not exceed 150 characters I would prefer not to write 150 spaces into a variable (because there are not guarantees that this length will be enough).

I will study all the versions and see what is the best solution for my needs.

Thank you all.

Saso

[foxidrive]

I would prefer not to write 150 spaces into a variable (because there are not guarantees that this length will be enough).

I'm unclear on the task - but a variable can have 8,000 characters and more, if you need to.

[Squashman]

Though in practice records do not exceed 150 characters I would prefer not to write 150 spaces into a variable (because there are not guarantees that this length will be enough).

Besides the limitations of 8192 for a variable in batch who really cares if it had a 2 million spaces. You can always SUBSTRING the variable for whatever length you want!

[miskox]

I would prefer not to write 150 spaces into a variable (because there are not guarantees that this length will be enough).

I'm unclear on the task - but a variable can have 8,000 characters and more, if you need to.

I meant that I don't want to have a variable in a .cmd with 150 spaces:

Code: Select all

set all_spaces=                               &rem 150 spaces are here

Though in practice records do not exceed 150 characters I would prefer not to write 150 spaces into a variable (because there are not guarantees that this length will be enough).

Besides the limitations of 8192 for a variable in batch who really cares if it had a 2 million spaces. You can always SUBSTRING the variable for whatever length you want!

Thanks. These are extreme situations.

Detailed explanation:

My 'database' file (actually .txt file) contains:

Code: Select all

123456789;123;somedata;somedata;somedata;somedata;somedata;

123456789 is file offset
123 is the record length

So when I have a record which record size is different from the current record length I must remove the data

Code: Select all

somedata;somedata;somedata;somedata;somedata;

with all spaces. It is important to have the exact number of spaces because the next record has the file offset in the record.

Original records (file offsets and record lengths are not correct - just an example):

Code: Select all

123456780;12;previous record;
123456789;123;somedata;somedata;somedata;somedata;somedata;
123456900;20;next record;

'Deleted' record (select them to see the spaces):

Code: Select all

123456780;12;previous record;
123456789;123;                                              
123456900;20;next record;

Thanks.
Saso

[dbenham]

Here is a small variation on Aacini's recommendation, using powers of two. The following will produce a space variable containing 1024 spaces, and delayed expansion is only temporarily enabled:

Code: Select all

set "space= "
setlocal enableDelayedExpansion
for /l %%N in (1 1 10) do set "space=!space!!space!"
endlocal & set "space=%space%"

This only needs to be done once. As others have suggested, from that point on you can simply use a substring expansion to get any length up to 1024. For example %space:-123% will yield 123 spaces.


Dave Benham

[Squashman]

Are the file offset and the record length included in the entire record length?

[Aacini]

So when I have a record which record size is different from the current record length I must remove the data

I don't understand the reason to do that. May I do some recommendations about your database?

• I don't understand why you store the offset of each record inside the record itself. The offsets are used to quickly locate a certain record via FilePointer.exe program, so they should be stored in a separated file that is used as index file to access the data one. The offsets of all lines in a file may be easily obtained with just findstr /O "^" theFile.txt, and FilePointer.exe may return the offset of the next record to process.
• The record lenght may be calculated via a subtraction of next offset minus current one, but it may be useful to store it in the record.
• Do you plan to re-use the deleted records? If so, how do you plan to identify them? It is inefficient to compare its contents vs. spaces. It is simpler to use a more direct method, like a "deleted record" mark; for example, store the record lenght with negative sign. If you do so:
• There is no need to store spaces in deleted records (just the "deleted" mark), unless you want to wipe-out sensible information.

viewtopic.php?f=3&t=5552&p=34051#p34051

Antonio

[miskox]

(Again sorry for a delay.)

Here is a small variation on Aacini's recommendation, using powers of two. The following will produce a space variable containing 1024 spaces, and delayed expansion is only temporarily enabled:

Code: Select all

set "space= "
setlocal enableDelayedExpansion
for /l %%N in (1 1 10) do set "space=!space!!space!"
endlocal & set "space=%space%"

This only needs to be done once. As others have suggested, from that point on you can simply use a substring expansion to get any length up to 1024. For example %space:-123% will yield 123 spaces.


Dave Benham

I will use this. I like this solution.

Are the file offset and the record length included in the entire record length?

This information is really not neccessary because all I wanted to know is how to get the desired number of spaces.

@Aacini (not to copy your long post):
I prefer this method of not using another 'index' file: I can search for desired record by different strings (logical AND). And when all the strings are found in ONE record then I process that record(s). I read this temporary file record by record and instantly get the record offset and it's length.

I don't reuse deleted records. I either modify/delete existing records or add new records. I also have a 'index rebuild' subprogram to recalculate record offsets and record lengths.

I must delete records I don't need (I cannot use 'deleted' mark) - the record that should be deleted is no longer valid. If I keep this information I can always find it (because I use FINDSTR with 1 (or more) input strings).

For example, actual file can look like this:

Code: Select all

        0; 95;_FIELD;STREET;LEVEL;APP_NUM;OWNER;LOCATION;SERIAL_NUM;PHONE;BUILDING_MGR;COMMENT;
       97;102;Field1;Brown street;+00;0001;John Doe;Hallway;34054791;1-800-1234567;Building Manager;#;
      201; 99;Field1;Brown street;+00;0001;John Doe;Room;34054792;1-800-1234567;Building Manager;#;
      302;104;Field1;Brown street;+00;0002;Susan Doe;Bathroom;34054700;1-800-2345678;Building Manager;#;
      408;103;Field1;Brown street;+00;0002;Susan Doe;Kitchen;34054701;1-800-2345678;Building Manager;#;

I use FINDSTR to find the record. Though the main 'key' in this file is SERIAL_NUMBER I can actually find the records in this way:

Let's say I want to find "John" for whom I know lives at "Brown street":

1. First I do a FINDSTR for %1: John into a temporary file
2. Then I do FINDSTR for %2: street from the temporary file into another temporary file
3. Rename another temporary file to the first temporary file and do more searches if neccessary. If not then I process this file record by record (of course with FOR /F to separate fields into variables etc...)
4. If I want to update/delete the record and if the new record length would not be the same I have to write all spaces when the actual data is (so I don't have trailing spaces in this record). When updating the data I then write the new record at the end of the file.

Working example was in another post (viewtopic.php?p=34025#p34025)

Thank you all.

Saso

[Squashman]

Sounds like you are slowing down your process by using extra FINDSTR commands and all those temp files.

[miskox]

Sounds like you are slowing down your process by using extra FINDSTR commands and all those temp files.

Currently things are working very well. I am sure you experts would find a better solution but everything is working great. I have some 23000 records so FINDSTR works very fast.

I will prepare another post with this problem (because it is not related to this original post).

Thanks.
Saso