Combine csv problem

[Sam85]

Hello,

I have a question and I am new with this type of programming.
When I use this code with small csv files, everything runs fine but if the csv files get bigger, nothing happens.

Is there a limit of data (rows, columns) inside a csv file, if I want to combine many csv files with the following code?

Looking forward to your reply.
Thanks and kind regards, Sam

Code: Select all

@echo on

ECHO Set working directory
pushd %~dp0

ECHO Deleting existing gesamt file
set "gesamt=Y:\verwaltung\schultes\Changes\TEST\gesamt.csv"
del "%gesamt%"

setlocal ENABLEDELAYEDEXPANSION
set cnt=1

for %%i in (*.csv) do (
if !cnt!==1 (
for /f "delims=" %%j in ('type "%%i"') do echo %%j >> "%gesamt%"
) else if %%i NEQ "%gesamt%" (
for /f "skip=1 delims=" %%j in ('type "%%i"') do echo %%j >> "%gesamt%"
)
set /a cnt+=1
)

[aGerman]

Untested:

Code: Select all

...

set "gesamt=Y:\verwaltung\schultes\Changes\TEST\gesamt"
if exist "%gesamt%.csv" del "%gesamt%.csv"
>"%gesamt%.tmp~" type nul

set "first=yes"
>>"%gesamt%.tmp~" (
  for %%i in (*.csv) do (
    if defined first (
      type "%%~i"
      set "first="
    ) else (
      <"%%~i" more +1
    )
  )
)

move "%gesamt%.tmp~" "%gesamt%.csv"

Steffen

[Sam85]

Good Morning Steffen,

I have tried your Code and it starts to work but then stucks somewhere, at least it looks like that.
Or does it just take a little longer?
I have put some screenshots on, with the folder and the DOS Console, to show what happend.
Thanks a lot, I am patient.

(I will continue in english now, didn't know about the community before thanks for the advice..sorry for my bad grammar or spelling)

Code: Select all

@echo on

ECHO Set working directory
pushd %~dp0

ECHO Deleting existing gesamt file
set "gesamt=Y:\verwaltung\schultes\Changes\TEST\gesamt"
if exist "%gesamt%.csv" del "%gesamt%.csv"
>"%gesamt%.tmp~" type nul

set "first=yes"
>>"%gesamt%.tmp~" (
  for %%i in (*.csv) do (
    if defined first (
      type "%%~i"
      set "first="
    ) else (
      <"%%~i" more +1
    )
  )
)

move "%gesamt%.tmp~" "%gesamt%.csv"

[aGerman]

I'm not sure what happens. It seems not even the first file was appended to the temporary file. That's a little strange because I'm not aware of a size limit for TYPE. When I'm back home this evening I'll try to reproduce this behavior. Currently I'm at work without having time for those test.

Steffen

PS. The preferred language at DosTips is English. There are people from all over the world joining. They are not able to understand your issues and thus, can't help out.

[miskox]

Is it possible that the MORE command displays

Code: Select all

-- More (3%) --

prompt after the page is full?

Maybe you can answer with pressing the SPACE key (or Q to quit and it should finish immediately I guess).

Help for more:

Code: Select all

    If extended features are enabled, the following commands
    are accepted at the -- More -- prompt:

    P n     Display next n lines
    S n     Skip next n lines
    F       Display next file
    Q       Quit
    =       Show line number
    ?       Show help line
    <space> Display next page
    <ret>   Display next line

Saso

[Sam85]

Hmm, i can't type anything...there is just a blinkin "_"...
can I add something like a Progressbar?

PS: now during my typing in this forum, i saw the new csv file is increasing or getting bigger (since I can't close the DOS Console, it still runs)...
so something like a progressbar or just a percentage view would be nice, if there is a possibilty.

PSS: now it stucks at 197 mb...

[Aacini]

Try this:

Code: Select all

@echo off
setlocal EnableDelayedExpansion

ECHO Set working directory
pushd "%~dp0"

ECHO Set gesamt file
set "gesamt=Y:\verwaltung\schultes\Changes\TEST\gesamt.out"

rem Get header from first file and count files
set "n=0"
set "header="
for %%i in (*.csv) do (
   if not defined header set /P "header=" < "%%i"
   set /A n+=1
)

(
echo %header%

set "i=0"
for %%i in (*.csv) do (

   set /A i+=1
   echo Processing file !i!/%n%: "%%i" > CON

   < "%%i" (

   rem Omit header
   set /P "header="

   rem Copy the rest
   findstr "^"

   )

)
) > "%gesamt%"

for %%f in ("%gesamt%") do move /Y "%%f" "%%~Nf.csv"

Antonio

[Sam85]

Hello Antonio,

do I really type it ".out"?

Try this:

Code: Select all

set "gesamt=Y:\verwaltung\schultes\Changes\TEST\gesamt.out"

Antonio

[Aacini]

Yes. You may use any extension you want as long as NOT be .csv! I make a small change in the code: I changed the last REN by a MOVE.

[Sam85]

Thanks Antonio It looks like it is working now...
but how do I get this out (see picture) or is this because of the last REN?

[Aacini]

Put "@ECHO OFF" at beginning!

[Sam85]

This is so exciting thanks lot...it works perfect.
By the way, can set a percentage countdown during the fusion between those files..like 1%, 2% etc?

[Aacini]

Ready!

The new code is in the same place of the original...

Antonio

[Sam85]

So amazing ...one last thing:

If I want to place the combined file in a other directory, what must be changed?

[aGerman]

Just change the path of "gesamt" and finally adjust the last line

Code: Select all

for %%f in ("%gesamt%") do move /Y "%%f" "%%~DPNf.csv"

Steffen