DosTips.com

Posted: **12 Jun 2016 16:19**

Hi, this is part of my code:

========================================
setlocal EnableDelayedExpansion
for /F "tokens=*" %%A in (All_csv.txt) do (
dir /b "%%A"* > "%%A".txt
)
for /F "tokens=*" %%A in ("%%A") do (
call :Delete_Last_Line '%%A'
)

:Delete_Last_Line %%A
set LINES=0
for /f "delims==" %%I in ("%%A") do (
set /a LINES=LINES+1
)
echo Total Lines : %LINES%
echo.
:: n = 1 , last n line will ignore
set /a LINES=LINES-1
call :PrintFirstNLine > "%%A".csv.txt
exit /b

:PrintFirstNLine
set cur=0
for /f "delims==" %%I in ("%%A") do (
echo %%I
::echo !cur! : %%I
set /a cur=cur+1
if "!cur!"=="%LINES%" goto EOF
)
exit /b

:EOF
========================================

I have a file named All_csv.txt and it contains a list of too many text files in the same path.
The idea of my script is to read all the lines from All_csv.txt file, create a new text file with the same name each row has and adding .txt and after that, delete last line in each text file created in there.

In my script, at the time I called subroutine, the error I got, said that the system cant find the file %A

The first FOR works ok, but at the time script jumps to the second FOR, it looks that the subroutine Delete_Last_Line can't find the file created in the first FOR.

I've tried in different ways to pass a variable and I can't make it work.
If I use in all the script a file name instead of a variable, the script works fine

Please could you take a look and let me know what I did wrong in my code?

Thank you in advance.

Posted: **12 Jun 2016 17:38**

mimismo wrote:I have a file named All_csv.txt and it contains a list of too many text files in the same path.

The obvious Q is "How these file names are separated? Are they stored on a new line each in All_csv.txt file?
Another Q is, why do you use DIR /B instead of ECHO in 1st FOR /F ?
Assuming the above, each new file would have only one line, the last one. If you delete it later from each, all these files will be empty.

Lets assume, each line in All_csv.txt contains several complete file paths. Now keep in mind, variable %%A only changes while 1st FOR /F loop is running. You need to include the 2nd FOR /F loop inside the 1st loop to enjoy %%A variety. Don't forget to use %%A.txt instead of %%A in the 2nd loop file name, and %%K as its variable to avoid mix-up. I didn't look further into your code, as this seems to be (the 1st set of) errors. :roll:

Its also a good idea to always post an exact error text. You also seems to exit 2nd function to :EOL without memory cleanup, which can make your Cmd window unusable for batch restarts.

Posted: **13 Jun 2016 10:32**

All those text files are separated in a new line each and none of the text files have one line, at least they have minimum 2 lines.

I've modified my code and nested all four steps needed in a FOR loops
Please take a look

=============================================
echo "Number 1
for /F "tokens=*" %%A in (All_csv.txt) do (
echo "Number 2"
dir /b "%%A"* > "%%A".txt
echo "%%A".txt
for /F "tokens=*" %%K in ("!%%A!".txt) do (
set LINES=0
for /f "delims==" %%I in ("!%%A!".txt) do (
set /a LINES=LINES+1
)
echo Total Lines : %LINES%
set /a LINES=LINES-1
echo %LINES%
set cur=0
for /f "delims==" %%L in ("!%%A"!.txt) do (
echo %%L
::echo !cur! : %%L
set /a cur=cur+1
if "!cur!"=="%LINES%" goto EOF
)
:EOF
)
)
=============================================
I echoed to debug my code and it exit at the time in into the first FOR.
I can see first echo, but nothing else.
The error I got is:
"was unexpected ) at this time."

Is there something I missed into my code?
Thank you

Posted: **13 Jun 2016 13:58**

You seems to miss quite a few things. :wink:

Pls post a sample of All_csv.txt content. Also, try using <Code> style to post code on the forum, multilevel indents for code blocks inside IF and FOR loops.

For debugging, open Cmd window and run your batch inside it. Don't enclose :eof inside the loop, I was merely talking about the 2nd FOR loop inside the 1st one, keep called functions outside the loop too if any used, count parenthesis. Generally, look through your code before posting, things like "%%A"* and "!%%A"!, delims==, ::echo ....

Posted: **14 Jun 2016 23:52**

mimismo wrote: for /f "delims==" %%L in ("!%%A"!.txt) do (

One ! is in the wrong place here. It is harder to debug code when we don't have the file that is being processed.

Posted: **15 Jun 2016 07:39**

Please let me explain in detail what I want to do.
I have hundred of files like this examples:
Each txt file contain a variable rows between one to ten every one in one line and none of them are blank rows and at least there is one row in each file
All those files contains values from monitoring tool that are storing data at variable intervals

Results from one day of monitoring are stored in "C:\Users\My User\My Path\Day one" and it is the same for every day except the path
"File Day One Server One 7.00-7.45.txt"
"7:00","25"
"7:30","33"
"7:45","20"

"File Day One Server One 9.13-19.21.txt"
"9:13","200"
"10:03","15"
"11:04","0"
"15:28","88"
"19:21","2"

"File Day One Server Two 0.00-3.22.txt"
"0:00","25"
"3:22","18"

"File Day One Server Two 5.14-5.14.txt"
"5:14","100"

So I need to get a chart for the whole day in excel for each server, one for Server One and another for Server Two.
Because of that, I need to concatenate all files from Server One in one single file to chart it and the same situation for the next servers.

1. Based on that, my idea to do this is list all files related to Server One and the same for the rest.
"File Day One Server One.txt"
"File Day One Server One 7.00-7.45.txt"
"File Day One Server One 9.13-19.21.txt"

"File Day One Server Two.txt"
"File Day One Server Two 0.00-3.22.txt"
"File Day One Server Two 5.14-5.14.txt"

2. List all the files in a single file
"All Files Day One.txt"
"File Day One Server One 7.00-7.45.txt"
"File Day One Server One 9.13-19.21.txt"
"File Day One Server Two 0.00-3.22.txt"
"File Day One Server Two 5.14-5.14.txt"

3. Concatenate the new files related to each server in order to get everything from this day in one file. In my case it will contain:
"File Day One Server One.txt"
"7:00","25"
"7:30","33"
"7:45","20"
"9:13","200"
"10:03","15"
"11:04","0"
"15:28","88"
"19:21","2"

"File Day One Server Two.txt"
"0:00","25"
"3:22","18"
"5:14","100"

4. Chart it.

Now I'm stuck in Step 3, because I can't concatenate it and I can't make it works.

My last code is:

Code: Select all

for /F "tokens=*" %%B in ("All Files Day One.txt"t) do (
   for /F "tokens=*" %%C in ("%%B") do (
      set Tem="%%C"
      call :JoinFiles
   )
)

:JoinFiles
echo Global Var Tem: %Tem%
set MyVar=%Tem%
for /F "tokens=*" %%D in (%Tem%) do (
   set %MyVar%=%MyVar%+"%%D"
)
   copy /a %MyVar% %MyVar%.csv
exit /b

Do you mind to give me a clue of what can do?
Thank you in advance

Posted: **15 Jun 2016 12:42**

Your task sounds quite straight forward and it's just the code to gather sets of files the sticking point.

It's not so hard to gather sets of filenames if they have a well defined format, and it's useful to know that format in advance.

Posted: **15 Jun 2016 14:37**

Thank you foxidrive

The main problem is that there are no names based in some logic for all files.
The only logic behind that is that for each server the monitoring tool generates a file ending with .csv and the next file ends with csv.1 and so on
Please find below the example of the names and each one is a file

For Server One:
SM R# MTPSM2-70.54.csv
SM R# MTPSM2-70.54.csv.1
SM R# MTPSM2-70.54.csv.2

For Server Two:
SME CALLs MA-15.54.csv
SME CALLs MA-15.54.csv.1
SME CALLs MA-15.54.csv.2
SME CALLs MA-15.54.csv.3

The content of the file "SME CALLs MAX-15.54.csv" is:
"06/02/2016 20:16:14","4"
"06/02/2016 20:16:24","3"
"06/02/2016 20:16:34","4"

And I should concatenate all files in reverse order for Server One and generates a new one, this is because old file for this polling have a name ending with a highest number and at the time I´ll chart it, the content should be sorted in order to the chart have a sense.
I mean, I should get something like this for Server One:

copy /a "SM R# MTPSM2-70.54.csv.2"+"SM R# MTPSM2-70.54.csv.1"+"SM R# MTPSM2-70.54.csv" SM R# MTPSM2-70.54.ALL.csv

Same thing for all the rest of servers monitoring tool is polling.

Thank you again

Posted: **15 Jun 2016 15:01**

I started with your four test files and based on your most recent post, this script should do step 1. for you. Did not see a purpose for step 2. By coincidence, Antonio recently addressed a similar need in Topic: Post subject: Re: Concatenation in for loop at http://www.dostips.com/forum/viewtopic.php?p=46969#p46969. This adapted easily to your step 3.

Code: Select all

   @echo off
    setlocal EnableExtensions EnableDelayedExpansion
    Set "My Path=%userprofile%\Scripts"
    Set "day=One" 
    Set "server=" 
   @echo.
    Dir "%My Path%\Day %day%\File Day %day% Server * *.txt"
    For /F "UseBackQ Tokens=5 delims= " %%S in (
        `Dir "%My Path%\Day %day%\File Day %day% Server * *.txt" /b` 
    ) do (
        IF /I "%%~S" NEQ "!Server!" ( 
            Set "Server=%%~S" 
           @echo.
            Dir "%My Path%\Day %day%\File Day %day% Server !server! *.txt" /B
            > "%My Path%\Day %day%\File Day %day% ServerList !server!.txt" ( 
            For /F "UseBackQ Tokens=* delims= " %%F in (
                `Dir "%My Path%\Day %day%\File Day %day% Server !server! *.txt" /B` 
            ) Do Echo."%My Path%\Day %day%\%%F") 
            > "%My Path%\Day %day%\File Day %day% Server !server!.txt" ( 
            for /F "UseBackQ tokens=* delims=" %%a in (
                "%My Path%\Day %day%\File Day %day% ServerList !server!.txt"
            ) do type %%a) 
           @echo.
            Echo.Contents: "%My Path%\Day %day%\File Day %day% Server !server!.txt"
            Type "%My Path%\Day %day%\File Day %day% Server !server!.txt"
        )
    )
    Exit /B

There are a few ECHOs to remove for a larger run. Customize the SETs at the beginning for your folder location. Here's a run output:

Code: Select all

C:\Users\Zani\Scripts>mon2csv

 Volume in drive C is Acer
 Volume Serial Number is 62EE-8C40

 Directory of C:\Users\Zani\Scripts\Day One

06/15/2016  12:38 PM                39 File Day One Server One 7.00-7.45.txt
06/15/2016  12:37 PM                68 File Day One Server One 9.13-19.21.txt
06/15/2016  12:36 PM                26 File Day One Server Two 0.00-3.22.txt
06/15/2016  12:35 PM                14 File Day One Server Two 5.14-5.14.txt
               4 File(s)            147 bytes
               0 Dir(s)  77,046,489,088 bytes free

File Day One Server One 7.00-7.45.txt
File Day One Server One 9.13-19.21.txt

Contents: "C:\Users\Zani\Scripts\Day One\File Day One Server One.txt"
"7:00","25"
"7:30","33"
"7:45","20"
"9:13","200"
"10:03","15"
"11:04","0"
"15:28","88"
"19:21","2"

File Day One Server Two 0.00-3.22.txt
File Day One Server Two 5.14-5.14.txt

Contents: "C:\Users\Zani\Scripts\Day One\File Day One Server Two.txt"
"0:00","25"
"3:22","18"
"5:14","100"

C:\Users\Zani\Scripts>

John A.

Posted: **15 Jun 2016 23:51**

mimismo wrote:The main problem is that there are no names based in some logic for all files.

Your examples show that the first hyphen in each filename is the end of the server name and this can be used to distinguish sets of servers. Is that the case with each server name?

Code: Select all

SM R# MTPSM2-
SME  CALLs MA-

Regarding sorting the files in reverse order - are there more than 10 of any set of files? What is the maximum number of files for any one server?

For Server One:
SM R# MTPSM2-70.54.csv
SM R# MTPSM2-70.54.csv.1
SM R# MTPSM2-70.54.csv.2

For Server Two:
SME CALLs MA-15.54.csv
SME CALLs MA-15.54.csv.1
SME CALLs MA-15.54.csv.2
SME CALLs MA-15.54.csv.3

And I should concatenate all files in reverse order

copy /a "SM R# MTPSM2-70.54.csv.2"+"SM R# MTPSM2-70.54.csv.1"+"SM R# MTPSM2-70.54.csv" SM R# MTPSM2-70.54.ALL.csv

Posted: **16 Jun 2016 08:35**

mimismo wrote:The content of the file "SME CALLs MAX-15.54.csv" is:
"06/02/2016 20:16:14","4"
"06/02/2016 20:16:24","3"
"06/02/2016 20:16:34","4"

And I should concatenate all files in reverse order for Server One and generates a new one, this is because old file for this polling have a name ending with a highest number and at the time I´ll chart it, the content should be sorted in order to the chart have a sense.

Stressing the order of concatenation leads to a contradiction in the order of the resulting data. Since each file contains ascending data, a reverse order concatenation will result a step order with chunks of ascending data followed by older chunks of ascending data. I strikes me that the file needs sorting and therefore the order of concatenation does not matter. That should make it easier.

PS You radically changed the filename format as well as the data content in a post that came in while I was typing my last response. The data content did not affect anything, but . . .

John A.

Posted: **16 Jun 2016 12:04**

Although more clarification is still required, I updated my code offering.

First, the time data in both versions does not appear to be sortable as it is. Your latest example does not contain a time before 10am, so I do not know if it is naturally sortable. Would 8:16:14 AM appear as 08:16:14 or as 8:16:14?. The date field is not sortable but can be ignored if the data is still batched in daily folders as you described. There are two logical places to add a converted sorted timestamp for this processing.

Foxidrive's question about the number of CSVs per server needs an answer. I suggest a pass of the data to a working folder renaming:

Code: Select all

SM R# MTPSM2-70.54.csv
SM R# MTPSM2-70.54.csv.1
SM R# MTPSM2-70.54.csv.2

to

Code: Select all

SM R# MTPSM2-70.54.csv.000
SM R# MTPSM2-70.54.csv.001
SM R# MTPSM2-70.54.csv.002

Then the reverse concatenation becomes natural. This is also a good place to introduce a usable timestamp.

In the meantime here is a structure using a new file format. I used old style data to test.

Code: Select all

   @echo off
:: Post subject: Re: Passing variable to subroutine
::Posted: Wed Jun 15, 2016 5:01 pm by thefeduke
::http://www.dostips.com/forum/viewtopic.php?p=47043#p47043
::Altered: Thu Jun 16, 2016 by thefeduke
    setlocal EnableExtensions EnableDelayedExpansion
    Set "My Path=%userprofile%\Scripts"
    Set "day=Day One"
    Set "server="
   @echo.
    Dir "%My Path%\%day%\*-*.csv*" /O-N 
      
    For /F "UseBackQ Tokens=1-3* delims=-." %%S in (
        `Dir "%My Path%\%day%\*-*.csv*" /O-N /B` 
    ) do (
        IF /I "%%~S" NEQ "!Server!" (
            Set "Server=%%~S"
            Set "ServTime=%%~T.%%~U"
           @echo.
      
::          Save list of files by server in descending order      
            > "%My Path%\%day%\!server!-!ServTime!.txt" (
            For /F "UseBackQ Tokens=* delims= " %%F in (
                `Dir "%My Path%\%day%\!server!-!ServTime!.csv*" /O-N /B`
            ) Do Echo."%My Path%\%day%\%%F")
      
::          Concatenate data in descending order from list of files
            > "%My Path%\%day%\!server!-%%~T.%%~U.ALL.csv" (
            for /F "UseBackQ tokens=* delims=" %%a in (
                "%My Path%\%day%\!server!-!ServTime!.txt"
            ) do Sort /R %%a)
      
           @echo.
            Echo.Contents: "%My Path%\%day%\!server!-!ServTime!.ALL.csv"
            Type "%My Path%\%day%\!server!-!ServTime!.ALL.csv"
        )
    )
     
    Exit /B

The last SORT statement could be replaced by a CALL to a subroutine to take care of the timestamps.

John A.

Posted: **17 Jun 2016 00:03**

Well, mimismo, I got carried away before the questions were answered. There are now workfiles ending in .nnn.csv just because the true extension is now more meaningful. My rudimentary time reformat has no effect on 8-character military time, but will add one missing leading zero if needed. I have assumed 0:00:01 as the worst case to correct. This time tested with very limited data in format: "06/02/2016 20:16:24","3" .

Code: Select all

   @echo off
:: Post subject: by mimismo Re: Passing variable to subroutine
::Posted: Wed Jun 15, 2016 5:01 pm by thefeduke
::http://www.dostips.com/forum/viewtopic.php?p=47043#p47043
::Altered: Fri Jun 17, 2016 by thefeduke
    setlocal EnableExtensions EnableDelayedExpansion
    Set "My Path=%userprofile%\Scripts"
    Set "day=Day One"
    Set "server="
   @echo.
Rem.Dir "%My Path%\%day%\*-*.csv*" /O-N

    For /F "UseBackQ Tokens=1-4* delims=-." %%S in (
        `Dir "%My Path%\%day%\*-*.csv*" /O-N /B`
    ) do (
        IF /I "%%~S" NEQ "!Server!" (
            Set "Server=%%~S"
            Set "ServTime=%%~T.%%~U"
::          Reformat suffix for workfiles
            Set "Ext=000%%~W"
            Set "Ext=!Ext:~-3!"
            Set "ServExt=!Ext!.%%~V"
           @echo.

::          Examine and group lists of files by server in descending order
            > "%My Path%\%day%\!server!-!ServTime!.txt" (
            For /F "UseBackQ Tokens=* delims= " %%F in (
                `Dir "%My Path%\%day%\!server!-!ServTime!.csv*" /O-N /B`
            ) Do (
::              create and save list of Work files by server in descending order
                Echo."%My Path%\%day%\!server!-!ServTime!.!ServExt!"
                > "%My Path%\%day%\!server!-!ServTime!.!ServExt!" (
                for /F "UseBackQ tokens=1* delims= " %%A in (
                    "%My Path%\%day%\%%F" 
                ) do (
::                  Reformat time field for sortablility
                    Set "Test=%%B" 
                    If "!Test:~1,1!" EQU ":" Set "Test=0!Test!"
                    Echo.%%A !Test!
                )) 
            ))

::          Concatenate sorted data in descending order from list of files
            > "%My Path%\%day%\!server!-%%~T.%%~U.ALL.csv" (
            for /F "UseBackQ tokens=* delims=" %%a in (
                "%My Path%\%day%\!server!-!ServTime!.txt"
            ) do Sort /R %%a)

           @echo.
            Echo.Contents: "%My Path%\%day%\!server!-!ServTime!.ALL.csv"
            Type "%My Path%\%day%\!server!-!ServTime!.ALL.csv"
        )
    )

    Exit /B

John A.

Posted: **18 Jun 2016 09:01**

Hi Team, thank you for all your hints.
Finally I was able to make works my script and I wanted to share with you it and I know this code easily can be tuned for you guys, now I'm feel satisfied by my first a bit complex script I've done.

Please let me explain in a brief what I did and in commented way, why I wrote every step:

Code: Select all

@echo off
setlocal EnableDelayedExpansion
:: Files generated by monitoring tool and they are garbage
del /q dirlock.bin
del /q rtmt.log

:: Monitoring tool generates files starting with the name of the server and ending with .csv, .csv1, .csv2 and so on until .csv10
:: This is because when I use dir command csv.10 is listed first even if it is the oldest file, so I must place at the end.
rename *.csv.10 *.91

:: Joined everything in a main text file.
:: In this step, I have in every row the name of each server to work with
dir /b *.csv > All.csv.txt

:: Working with the content of the main text file in order to get a list in a .csv files starting with the name of the each server
:: Each new csv file has the content of all related files for each server.
for /F "tokens=*" %%A in (All.csv.txt) do (
   :: listed in reverse order since the beginning in order to avoid further sorting.
   dir /b /o-n "%%A"* > "%%A".All.csv
   :: dir listing adds the redirected output in a new row and in each file it must be removed
   findstr /V "csv.All.csv" "%%A".All.csv > "%%A".csv
   rename "%%A".csv "%%A".txt
   del "%%A".All.csv
)

:: This step is used to remove the rows containing certain data that doesn't make sense for charting.
for /F "usebackq tokens=*" %%B in (All.csv.txt) do (
   for /F "usebackq tokens=*" %%C in ("%%B.txt") do (type "%%C">>"%%B.A1.csv")
   findstr /V "PDH" "%%B".A1.csv > "%%B".A2.csv
   del "%%B".A1.csv
)
:: Removing garbage
del /q *.csv.txt
:: This file contains in each row only the name of the files related to each server now with removed content
dir /b *csv.A2.csv > All.csv2.txt
:: Rolling back the original name for affected files
rename *.csv.91 *.10
:: In this step I must remove double quotes and exchange comma by semi colon
:: Old fashion:  "06/14/2016 11:54:23","24"
:: New fashion:  11:54:33;24
for /F "usebackq tokens=*" %%D in (All.csv2.txt) do (
   for /F "usebackq tokens=*" %%E in ("%%D") do (
      set Name=%%E
      set Name
      set Name=!Name:~1,-1!
      set Name
      set Name=!Name:~10!
      set Name
      set Name=!Name:"=!
      set Name
      set Name=!Name:,=;!
      set Name
      echo !Name! >> "%%D.All.csv"
   )
   del /q "%%D".A2.csv
)
:: Removing garbage
del /q All.csv2.txt
exit /b

The drawback is that I must run only once, but it works now and I can chart it in Excel.
No I must starting climbing the mountain to understand a different way of programming.
Again thank you all you guys who helped me to understand the basics
Sincerely