how to replace all occurrences of ;;;; with ; in a string

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: how to replace all occurrences of ;;;; with ; in a strin

#16 Post by dbenham » 31 May 2012 16:18

Squashman wrote:And Technically you can have 31 Tokens in a single FOR LOOP and a total of 65 Unique variables with nested For Loops. It is undocumented but it works apparently.

Actually there are only 3 byte codes that cannot be used as FOR variables: 0x00, 0x0D and 0xFF.

With careful planning, you should be able to have concurrent access to as many as 245 tokens within a single line of text by using 9 nested FOR /F loops. I pity the fool that tries to actually do that though :lol:

See http://stackoverflow.com/a/8520993/1012053 for more info.

Another thing that may blow you away - the FOR loops don't have to be strictly nested. When you CALL out of a FOR loop, the FOR variables are not in context within the sub-routine. But within a FOR loop inside the sub-routine the earlier FOR variables are suddenly visible again :shock: :D
I have a couple dostips posts some where that show this, but I can't find them at the moment.


Dave Benham

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: how to replace all occurrences of ;;;; with ; in a strin

#17 Post by foxidrive » 31 May 2012 19:25

dbenham wrote:
timbertuck wrote:the max char's for a variable is 8192 bytes, so theoretically one could add 5 add'l "if defined string set..." to reach that limit, is there a way to put the "if defined" in a for /l loop (to make the code more concise)?


Of course there is - just take my original code, and change the 5 to a 13 in the FOR loop. The code to strip leading and trailing commas is already there, as are all the necessary IF DEFINED statements.


The only drawback is that you lose a character - you cannot have ! characters in the input data.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: how to replace all occurrences of ;;;; with ; in a strin

#18 Post by dbenham » 31 May 2012 22:09

foxidrive wrote:The only drawback is that you lose a character - you cannot have ! characters in the input data.
Not an issue here. Enabling delayed expansion only causes expansion corruption of FOR variables containing !. But my original code has only numbers in the FOR variable, and it is never expanded anyway. The Squashman method only has commas in the FOR variable. The only value that might have ! is the string environment variable, but there is no problem expanding that using !string!.

The initial population of the string variable could be an issue, but for me, that is outside the scope of the problem. And there are multiple methods to solve that any way.

The normal expansion method you provided has its share of problems.

A string value like "Hello&Goodbye",,This&That will give normal expansion fits, whereas delayed expansion has no problem.


Dave Benham

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: how to replace all occurrences of ;;;; with ; in a strin

#19 Post by foxidrive » 01 Jun 2012 00:02

dbenham wrote:A string value like "Hello&Goodbye",,This&That will give normal expansion fits, whereas delayed expansion has no problem.


Ahhh, but your code won't work with embedded quotes with ampersands either. I was merely commenting upon ! characters.

Broken code follows:

Code: Select all

@echo off
setlocal enableDelayedExpansion
set "str=,,,"Hello&Goodbye",,This&That,value2 opt,,,value3,,value4 opt,,,value5,,,,,"
if defined str (
  for /l %%N in (1 1 5) do set "str=!str:,,=,!"
  if "!str:~0,1!"=="," set "str=!str:~1!"
  if defined str if "!str:~-1!"=="," set "str=!str:~0,-1!"
)
set str
pause

jeb
Expert
Posts: 1055
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: how to replace all occurrences of ;;;; with ; in a strin

#20 Post by jeb » 01 Jun 2012 01:30

Hi foxidrive,

but this fails only because the definition of the string is wrong.

It should be:

Code: Select all

set "str=,,,"Hello^&Goodbye",,This&That,value2 opt,,,value3,,value4 opt,,,value5,,,,,"

The rest of the code is bullet proof.

Btw. There exists another way of removing multiple commas without any loop.

Code: Select all

@echo off
setlocal enableDelayedExpansion
set LF=^


rem ** Two empty lines are required
set "str=,,,"Hello^&Goodbye",,,,,,,,,,,,,,,,This&That,value2 opt,,,value3,,value4 opt,,,value5,,,,,"
if defined str (
   for %%L in ("!LF!") DO (
      rem Add a linefeed "," -> "<LF>,"
      echo #%%~L#
      set ^"str=!str:,=%%~L,!"

      rem replace ",<LF>"->"" (empty)
      set ^"str=!str:,%%~L=!"

      rem replace ",<LF>"->"" (empty)
      set ^"str=!str:%%~L,=,!"
   )
)
set str



jeb

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: how to replace all occurrences of ;;;; with ; in a strin

#21 Post by foxidrive » 01 Jun 2012 02:41

jeb wrote:The rest of the code is bullet proof.


It is not bulletproof jeb. It will not handle ! characters. That is all I said. Here's an example.


Code: Select all

@echo off
setlocal enableDelayedExpansion
set "str=,,,!Hello Goodbye!,,This&That,value2 opt,,,value3,,value4 opt,,,value5,,,,,"
if defined str (
  for /l %%N in (1 1 5) do set "str=!str:,,=,!"
  if "!str:~0,1!"=="," set "str=!str:~1!"
  if defined str if "!str:~-1!"=="," set "str=!str:~0,-1!"
)
set str
pause



I don't know about any of you but I don't use % or ^ in my filenames but I do use ! hence I am aware of the limitation - it has to be spelled out to people asking for help because it is a common punctuation character.

jeb
Expert
Posts: 1055
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: how to replace all occurrences of ;;;; with ; in a strin

#22 Post by jeb » 01 Jun 2012 04:19

It will handle ! characters :)
But in your string aren't any exclamation marks left!

Try this instead

Code: Select all

@echo off
set "str=,,,!Hello Goodbye!,,This&That,value2 opt,,,value3,,value4 opt,,,value5,,,,,"

setlocal enableDelayedExpansion
set LF=^


rem ** Two empty lines are required

if defined str (
   for %%L in ("!LF!") DO (
      rem Add a linefeed "," -> "<LF>,"
      echo #%%~L#
      set ^"str=!str:,=%%~L,!"

      rem replace ",<LF>"->"" (empty)
      set ^"str=!str:,%%~L=!"

      rem replace ",<LF>"->"" (empty)
      set ^"str=!str:%%~L,=,!"
   )
)
set str


You loose it in the first set statement, obviously the rest will not insert them later.

But normally you would read the content from a file with SET/p or FOR/F and then you can handle them also without problems.

jeb

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: how to replace all occurrences of ;;;; with ; in a strin

#23 Post by dbenham » 01 Jun 2012 05:21

Similar to what jeb said - the problem isn't in the algorithm. It is in the creation of the test variable while delayed expansion is on. ! must be escaped, and if ! exists, then any ^ must also be escaped in a special way. The escape pattern changes depending on whether the character is quoted. The usual cast of unquoted special characters must also be escaped in the usual way.

Code: Select all

quoted normal special character.. &
quoted ! ........................ ^!
quoted ^ without presence of !... ^
quoted ^ in presence of !........ ^^
unquoted normal special character ^&
unquoted !....................... ^^!
uquoted ^ without presence of !.. ^^
unquoted ^ in presence of !...... ^^^^

desired test values:
,,,"^Hello & Goodbye!",,^This & That!,,,
,,,"^Hello & Goodbye",,^This & That,,,

desired results:
"^Hello & Goodbye!",^This & That!
"^Hello & Goodbye",^This & That

Code: Select all

@echo off
setlocal enableDelayedExpansion
echo Test with ^^!
set string=,,,"^^Hello & Goodbye^!",,^^^^This ^& That^^!,,,
call :collapseCommas
echo(
echo Test without ^^!
set string=,,,"^Hello & Goodbye",,^^This ^& That,,,
call :collapseCommas
exit /b

:collapseCommas
set string
:: Bullet-proof comma collapsing begins here!
if defined string (
  for /l %%N in (1 1 13) do set "string=!string:,,=,!"
  if "!string:~0,1!"=="," set "string=!string:~1!"
  if defined string if "!string:~-1!"=="," set "string=!string:~0,-1!"
)
:: comma collapsing ends here!
set string
exit /b

I didn't bother showing Squashman's algorithm, but once the string value is set properly, both algorithms are bullet proof.


Dave Benham
Last edited by dbenham on 01 Jun 2012 08:55, edited 1 time in total.

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: how to replace all occurrences of ;;;; with ; in a strin

#24 Post by foxidrive » 01 Jun 2012 05:33

jeb wrote:But normally you would read the content from a file with SET/p or FOR/F and then you can handle them also without problems.
jeb


The OP was not told how to process the code, and so my comment is still valid.

dbenham wrote:I didn't bother showing Squashman's algorithm, but once the string value is set properly, both algorithms are bullet proof.
Dave Benham


Your code as posted does not handle ! characters. I made the comment that it doesn't and you're arguing about the algorithm.

So try this.

Code: Select all

@echo off
> file.txt echo ,,,!value1!,value2 opt,,,value3,,value4 opt,,,value5,,,,,
setlocal enableDelayedExpansion
for /f "delims=" %%a in (file.txt) do (
set str=%%a
if defined str (
  for /l %%N in (1 1 5) do set "str=!str:,,=,!"
  if "!str:~0,1!"=="," set "str=!str:~1!"
  if defined str if "!str:~-1!"=="," set "str=!str:~0,-1!"
)
echo !str!
)
pause


Output

Code: Select all

value2 opt,value3,value4 opt,value5

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: how to replace all occurrences of ;;;; with ; in a strin

#25 Post by dbenham » 01 Jun 2012 07:33

I did not set the value of the string using a FOR loop did I. Nor did the OP specify how the variable is to be initially set. The comma collapsing code I posted absolutely works in all cases.

I do not consider how the string is initially set to be part of the original question.

FACT - given any string stored in an environment variable, the algorithms I've posted properly handle the collapsing of the commas. How the string is initially set is NOT part of the algorithm.

I'm not saying that setting the initial value is trivial in all cases. I already stated in an earlier post:
dbenham wrote:The initial population of the string variable could be an issue, but for me, that is outside the scope of the problem. And there are multiple methods to solve that any way.

The same cannot be said for any algorithm that relies on normal expansion without the use of delayed expansion. I do not believe it is possible to properly collapse commas for all possible string values unless delayed expansion is used.

To address your last situation that "doesn't work" - The solution is simple. And it does not change how the commas are collapsed.

Code: Select all

@echo off
> file.txt echo ,,,!value1!,value2 opt,,,value3,,value4 opt,,,value5,,,,,
setlocal disableDelayedExpansion
for /f "delims=" %%a in (file.txt) do (
  set str=%%a
  setlocal enableDelayedExpansion
  REM Bullet-proof comma collapsing begins here!
  if defined str (
    for /l %%N in (1 1 13) do set "str=!str:,,=,!"
    if "!str:~0,1!"=="," set "str=!str:~1!"
    if defined str if "!str:~-1!"=="," set "str=!str:~0,-1!"
  )
  REM comma collapsing ends here!
  echo !str!
  endlocal
)
pause

How would you do the above without delayed expansion? Not only would it not properly handle all possible values, but you would have to use at least one CALL statement. And CALL is SLOW. If the file is large then the delayed expansion solution will be MUCH faster.

There is also a solution if you need to disable delayed expansion when finished and preserve the value across the endlocal barrier. Yet again, the comma collapsing algorithm does not change.

Code: Select all

@echo off
> file.txt echo ,,,!value1!,value2 opt,,,value3,,value4 opt,,,value5,,,,,
setlocal disableDelayedExpansion
for /f "delims=" %%a in (file.txt) do (
  set str=%%a
  setlocal enableDelayedExpansion
  REM Bullet proof comma collapsing begins here!
  if defined str (
    for /l %%N in (1 1 13) do set "str=!str:,,=,!"
    if "!str:~0,1!"=="," set "str=!str:~1!"
    if defined str if "!str:~-1!"=="," set "str=!str:~0,-1!"
  )
  REM comma collapsing ends here!
  echo !str!
  for /f "eol=, delims=" %%b in ("!str!") do (
    endlocal
    set "str=%%b"
  )
)
echo %str%
pause

It is even possible to encapsulate the algorithm in a subroutine that can accept any string variable with absolutely any value, collapse the commas, and return the correct value across the endlocal barrier without worrying about whether the routine was called with delayed expansion enabled or disabled. Or for ultimate performance, the algorithm could be encapsulated in a macro that does the same. But that is way beyond the scope of the original question. The techniques to do this are the same regardless what algorithm you are trying to encapsulate, and the techniques have been extensively documented in other posts.


Dave Benham
Last edited by dbenham on 01 Jun 2012 08:53, edited 2 times in total.

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: how to replace all occurrences of ;;;; with ; in a strin

#26 Post by foxidrive » 01 Jun 2012 07:44

dbenham wrote:The code I posted absolutely works in all cases.


That is not so. See the previous post.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: how to replace all occurrences of ;;;; with ; in a strin

#27 Post by dbenham » 01 Jun 2012 08:26

foxidrive - Did you carefully read my entire post :?: :?: :?

By code, I meant the portion that actually deals with collapsing of the commas. (edit made in previous post)

I think we are done here.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: how to replace all occurrences of ;;;; with ; in a strin

#28 Post by dbenham » 01 Jun 2012 08:41

@jeb - interesting non looping algorithm. But you assume the string does not contain <LF> :wink:

I believe I have a bullet-proof variation that avoids using any awkward characters like <LF> and supports all possible string values. Though I'm not convinced I've done enough testing.

Code: Select all

@echo off
setlocal enableDelayedExpansion
set string=,,,"^^Hello & Goodbye^!",,^^^^This ^& That^^!,,,@,,,@#,,,#@,,,
set string

:: bullet-proof (I hope) comma collapsing begins here!
if defined string (
  set "string=!string:@=@@!"
  set "string=!string:#=@#!"
  set "string=!string:,=#,!"
  set "string=!string:,#=!"
  set "string=!string:#,=,!"
  set "string=!string:@#=#!"
  set "string=!string:@@=@!"
  if "!string:~0,1!"=="," set "string=!string:~1!"
  if defined string if "!string:~-1!"=="," set "string=!string:~0,-1!"
)
:: comma collapsing ends here!

set string

The code above requires 7 search and replaces, the same as the modified Squashman technique. So I don't think it has any advantage over it, especially if you flatten out the Squashman technique.

Code: Select all

@echo off
setlocal enableDelayedExpansion
set string=,,,"^^Hello & Goodbye^!",,^^^^This ^& That^^!,,,@,,,@#,,,#@,,,
set string

:: bullet-proof comma collapsing begins here!
if defined string (
  set "string=!string:,,,,,,,,=,!"
  set "string=!string:,,,,,,,=,!"
  set "string=!string:,,,,,,=,!"
  set "string=!string:,,,,,=,!"
  set "string=!string:,,,,=,!"
  set "string=!string:,,,=,!"
  set "string=!string:,,=,!"
  if "!string:~0,1!"=="," set "string=!string:~1!"
  if defined string if "!string:~-1!"=="," set "string=!string:~0,-1!"
)
:: comma collapsing ends here!

set string

If batch supported variables larger than 8k then your modified algorithm would definitely be advantageous.

Edit - One disadvantage of the jeb technique (modified or not) is that it is limited in the size of the variable it can process, and the limit depends on the number of characters that have to be doubled in width. The problem is the string must expand before it shrinks. So the original jeb method can only handle a string of nothing but commas that is ~4k in lenth. The modified version can only handle ~4k length if the string contains only comma, @ and #.

Dave Benham
Last edited by dbenham on 01 Jun 2012 09:29, edited 2 times in total.

timbertuck
Posts: 76
Joined: 21 Dec 2011 14:21

Re: how to replace all occurrences of ;;;; with ; in a strin

#29 Post by timbertuck » 01 Jun 2012 08:53

fascinating! thanks for all of the input on this question..

as for the issues described above, my string only contains "alphanumeric" characters with the exception of one variable (which could be a reserved character but i account for that in my program).

i do have a question for jeb, could you explain your code to me, i can understand some of it, but the rest i am lost. i get the LF portion, but the set strings i don't get. thanks!

Code: Select all

@echo off
set "str=,,,!Hello Goodbye!,,This&That,value2 opt,,,value3,,value4 opt,,,value5,,,,,"
setlocal enableDelayedExpansion
set LF=^


rem ** Two empty lines are required

if defined str (
   for %%L in ("!LF!") DO (
      rem Add a linefeed "," -> "<LF>,"
      echo #%%~L#
      set ^"str=!str:,=%%~L,!"

      rem replace ",<LF>"->"" (empty)
      set ^"str=!str:,%%~L=!"

      rem replace ",<LF>"->"" (empty)
      set ^"str=!str:%%~L,=,!"
   )
)
set str

MrKnowItAllxx
Posts: 43
Joined: 20 Mar 2012 20:53

Re: how to replace all occurrences of ;;;; with ; in a strin

#30 Post by MrKnowItAllxx » 01 Jun 2012 12:06

Call me lazy, but I didn't want to read 2 pages of comments, so I'm posting this solution w/o knowing if it has been suggested already

Code: Select all

@echo off
setlocal enabledelayedexpansion
set "string=,,,value1,value2 opt,,,value3,,value4 opt,,,value5,,,,,"
for /l %%a in (1,1,8) do set "string=!string:,,=,!"
set "string=!string:~1,-1!"
echo %string%
pause


as long as none of the occurences have more than 2^8 commas between them, it should solve the problem
Also, I am assuming none of the strings have commas in them because they are delimited by commas (obviously :P)

Post Reply