Performance Issues with Code

[Aacini]

Inspired by @Aacini's solution, some pseudo-code as I dont have the time and experience to convert this to proper code:

Code: Select all

Instead of month n=1..12, set month= 2^^n -1 (1.. 4095)
set /a min=4095, max=0
For loop:
  set /a min="min & month", max="max | month"

After the for-loop, convert 2^^n - 1 back to n.
 

Might be faster ..

This is a good idea!

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem Empty environment
(
   for /F "delims==" %%a in ('set') do set "%%a="
   set "ComSpec=%ComSpec%"
)

rem Define the month elements with a bit set in the month position, that is:
rem Bits: 12 11 10  9  8  7  6  5  4  3  2  1   Value:
rem Jan:   0  0  0  0  0  0  0  0  0  0  0  1 = 2
rem Feb:   0  0  0  0  0  0  0  0  0  0  1  0 = 4
rem ...
rem Dec:   1  0  0  0  0  0  0  0  0  0  0  0 = 4096

set /A "i=0, j=100"
for %%a in (January February March April May June July August September October November December) do (
   set /A "i+=1, j+=1, bit=1<<i"
   set "month=%%a"
   set /A "m%%a=bit, m!month:~0,3!=bit, m!i!=bit, m!j:~1!=bit"
)

SET "zBits=0"
FOR /F "skip=1 USEBACKQ tokens=2 delims=|" %%a in ("test.txt") DO set /A "zBits|=m%%~a"

rem Convert accumulated bit positions back to months numbers: first bit for Min, last bit for Max
set "MINM="
for /L %%b in (1,1,12) do (
   set /A "bit=1<<%%b & zBits"
   if !bit! neq 0 (
      set "MAXM=%%b"
      if not defined MINM set "MINM=%%b"
   )
)

if %MAXM% lss 10 set "MAXM=0%MAXM%"
if %MINM% lss 10 set "MINM=0%MINM%"

echo Min: %MINM%
echo Max: %MAXM%

Antonio

PS - Please post the timing of my two versions...

[Eureka!]

I'm getting between 60 and 80 seconds for a million rows for this, likely because two set statements are being run every single iteration.

Thanks for testing (and writing the code, of course)!

Lesson learned: Comparing variables is "cheaper" than setting variables. Good to know!

Code: Select all

[...]

Clever!!

[ShadowThief]

PS - Please post the timing of my two versions...

I ran both versions ten consecutive times with the same million-line data file.

Version 1:

Code: Select all

Start:    14:53:15.67
End:      14:54:17.30
Duration: 61.6 seconds

Start:    14:54:17.31
End:      14:55:19.49
Duration: 62.2 seconds

Start:    14:55:19.49
End:      14:56:21.07
Duration: 61.6 seconds

Start:    14:56:21.07
End:      14:57:22.52
Duration: 61.5 seconds

Start:    14:57:22.53
End:      14:58:24.05
Duration: 61.5 seconds

Start:    14:58:24.05
End:      14:59:25.51
Duration: 61.5 seconds

Start:    14:59:25.51
End:      15:00:27.02
Duration: 61.5 seconds

Start:    15:00:27.02
End:      15:01:29.54
Duration: 62.5 seconds

Start:    15:01:29.54
End:      15:02:37.40
Duration: 67.9 seconds

Start:    15:02:37.41
End:      15:03:41.34
Duration: 63.9 seconds

Average Duration: 62.6 seconds

Version 2:

Code: Select all

Start:    14:37:13.79
End:      14:37:47.59
Duration: 33.8 seconds

Start:    14:37:47.59
End:      14:38:21.38
Duration: 33.8 seconds

Start:    14:38:21.38
End:      14:38:55.22
Duration: 33.8 seconds

Start:    14:38:55.22
End:      14:39:30.02
Duration: 34.8 seconds

Start:    14:39:30.03
End:      14:40:05.44
Duration: 35.4 seconds

Start:    14:40:05.45
End:      14:40:40.72
Duration: 35.3 seconds

Start:    14:40:40.72
End:      14:41:15.61
Duration: 34.9 seconds

Start:    14:41:15.61
End:      14:41:50.93
Duration: 35.3 seconds

Start:    14:41:50.94
End:      14:42:25.61
Duration: 34.7 seconds

Start:    14:42:25.61
End:      14:43:01.46
Duration: 35.8 seconds

Average Duration: 34.8 seconds

[SIMMS7400]

WOW! I'm going to play around with these solutions and see what works best.

It looks like Shadow's and Atonio's solution are comparable? I forgot how fast Shadow's code ran so i will need to do some runs times...

[SIMMS7400]

Hello Gents -

With a file of about 700k lines, Antonio's solution is about 30 seconds faster. Pretty significant difference, but both are still unbelieveavly quick.

Thank you again to you both for these solutions!!

[ShadowThief]

I'd love to see the times you ended up with, because all of my tests have my code run about 10 seconds faster than anything else that's been posted so far.

[SIMMS7400]

Here are the results:

Code: Select all

Atonio's Solution 
Start Time : 19:54:44.36
End Time : 19:56:33.82
Shadow's Solution 
Start Time : 19:57:22.91
End Time : 19:58:54.68

File Size : 797761 lines

I stand corrected, Shadow's solution is quicker! My apologies for the incorrect message earlier, I was trying to time things "off the hip" on the fly.

[Aacini]

Which ShadowThief's solution is faster than which Antonio's solution? I wrote two solutions. The main loop in the second one is this:

Code: Select all

FOR /F "skip=1 USEBACKQ tokens=2 delims=|" %%a in ("test.txt") DO set /A "zBits|=m%%~a"

It is hard to me to belive that such a loop is slower than this one:

Code: Select all

FOR /F "skip=1 usebackq tokens=2 delims=|" %%a IN ("test.txt") DO ( 
    IF !month_val[%%~a]! GTR !MAXM! SET "MAXM=!month_val[%%~a]!"
    IF !month_val[%%~a]! LSS !MINM! SET "MINM=!month_val[%%~a]!"
)

Could you test my second code changing zBits with bits please? Thanks...

Antonio