Problem with escaping characters

[Quisquose]

A friend (whom I no longer have contact with) created a batch file for me that converts playlists between different formats.

I've been using it for ages and it works really great, except for when the playlist filename has characters like ampersands or apostrophes etc. in it (in which case the script breaks).

I'm no good with command line stuff, but I would have at least attempted to fix it by myself if it were a fairly simple script, but it's quite complicated, so the chances are I'd only end up make things worse with my trial-and-error approach.

I'm assuming that all it needs is certain characters to be 'escaped' in the parts of the batch that deal with file names (which I imagine would be fairly easy for someone experienced).

If there's anyone who is willing to take a look at the batch for me, then I'd be most grateful.

Thanks.

[penpen]

A sample playlist file (or more if needed) with problematic characters and the full command you are using when "things go wrong" would be nice.

penpen

[Quisquose]

Hi, thanks for your reply.

Just to clarify, the issue is not with characters inside the playlist (track names) they all work fine regardless of which characters are included. The issue is with the file name of playlist itself.

I seem to recall discussing this with my friend at the time that he was creating the script, and he included something that ensured all sorts of characters could appear in the individual track entries (within the playlist) without any problem. However, the same does not appear to have been done for the playlist's own file name.

I have attached two sample playlists; one containing an ampersand in the name, the other containing parentheses.

If I try to convert the one with the ampersand, I see the following error in the command window:

Code: Select all

Bass.m3u is not recognized as an internal or external command, operable program or batch file.
ERROR: The file C:\Users\Quisquose\Desktop\Drum  does not exist

If I try to convert the one with parentheses, then the console window flashes for a fraction of second, but I can't see what it says. In both cases no converted playlist file is output.

The command that I am using is: "C:\Users\xx\Files\Documents\playlist-converter-0.4.cmd" m2p inplace "%1"

The "inplace" command is just whether the original should be deleted or not after the conversion has been made. The m2p sets which format should be converted to which (in this case, m3u > pls) so it calls that particular section of the code. But obviously if I am converting in the other direction (pls > mp3) then I would use the "p2m" command.

I never actually type these commands, because I made a Registry key and created a context-menu just for those specific playlist file types. So I just right-click on a playlist and choose 'Convert Playlist' from the context menu (which runs a command similar to the one shown above).

I can multiple select many playlists and right-click and convert in one step, and they all convert correctly except for those with ampersands and parentheses in their file names. I originally thought there were other characters that were causing a problem too, but I've just tried again using playlists with apostrophes and other symbols (that are legal for file names) and they all worked.

I'm guessing it's because & and () have special meaning in batch files.

This got me thinking that maybe a playlist that had a % symbol in its name might also present a problem (seeing as that's another character that batch files use). When I tried it out just now, it did convert, but in this instance the file name was altered and the % symbol was stripped out (instead of maintaining the original file name like it should).

I have therefore added an additional sample playlist that contains a % in the file name (although you could just as easily rename one of the others to add a % sign to its name).

[Compo]

I've taken a look at your script and made a few small changes, mainly the addition of double quotes.

I have not and can not test the script, and hope that I haven't unintentionally broken something.

Give it a try and post feedback.

[aGerman]

You would need to completely rewrite the code. At least as long as delayed expansion is set for the whole script exclamation marks would always cause errors.
Also I'm certain that the file names of the play list are UTF-16 encoded (as every file name on Windows) which is doomed to failure if they are processed from the output of other commands and they contain characters that can't be represented by the currently used code page. The content of the m3u files will be encoded in the default ANSI code page. So at least I'd suggest to change the code page of the script process to the ANSI code page to avoid failures.

Steffen

[Quisquose]

Thank you, Compo.

Playlists with & or ( ) in the file name now work correctly without error, but the playlist with the % in its file name still has it stripped in the output file name.

However, while I do have numerous playlists with ampersands and parentheses in their names, I don't have any with percentage signs; that was just an example that I threw in at the last minute when it occurred to me that this symbol would likely be affected too.

So while the issue has not been totally resolved, in terms of the constant conversion back and forth that I do day-to-day (in order to use my playlists on my different devices) it makes a big difference, because now all of my files convert correctly in one go without error.

If, in future, I ever did have a playlist with a % sign (or an exclamation mark for that matter) then I would just have to do what I have had to do in the past - i.e. check which files didn't convert and then temporarily rename them for conversion (to remove the problem characters) before manually renaming the output file(s) back to the correct original file name. It's somewhat of a nuisance, but hardly the end of the world, and not something that I would encounter at present.

Thanks again.

[penpen]

The issue with the percentage sign is caused by using call (call :CREATE_EMPTY_PLA "!_newfile!").
You could avoid this, by passing the name of the variable containing the string, instead of the string itself.

My solution were to replace these lines (without the line number and double colon):

Code: Select all

 91:  if [%3]==[] (
 98:  set _file=%~3
111:  set _attr=%~a3
124:  call :GET_NEW_FILENAME "!_file!"
150:  if "!_direction!"=="m2p" call :CREATE_EMPTY_PLA "!_newfile!"
152:    if "!_delete!"=="true" call :DELETE_EMPTY_PLA "!_file!"
187:  if [%3]==[] (
194:  set _inputdir=%~f3
227:      call :GET_NEW_FILENAME "!_file!"
251:        if "!_direction!"=="m2p" call :CREATE_EMPTY_PLA "!_newfile!"
265:      call :GET_NEW_FILENAME "!_file!"
288:        if "!_direction!"=="m2p" call :CREATE_EMPTY_PLA "!_newfile!"

with the following lines:

Code: Select all

 91:  if "%~3"=="" (
 98:  set "_file=%~3"
111:  set "_attr=%~a3"
124:  call :GET_NEW_FILENAME "_file"
150:  if "!_direction!"=="m2p" call :CREATE_EMPTY_PLA "_newfile"
152:    if "!_delete!"=="true" call :DELETE_EMPTY_PLA "_file"
187:  if "%~3"=="" (
194:  set "_inputdir=%~f3"
227:      call :GET_NEW_FILENAME "_file"
251:        if "!_direction!"=="m2p" call :CREATE_EMPTY_PLA "_newfile"
265:      call :GET_NEW_FILENAME "_file"
288:        if "!_direction!"=="m2p" call :CREATE_EMPTY_PLA "_newfile"

And also replace the following functions with these versions:

Code: Select all

rem -----------------------------------
:GET_NEW_FILENAME
rem -----------------------------------
rem based on _direction
rem generate name of new file from old

if "!_direction!"=="m2p" (
  rem from .m3u to .pla.refs
  rem if original filename has .m3u extension, strip it,
  rem and append .pla.refs; if not just append .pla.refs
  
  set "_newfile=!%~1!;"
  set "_newfile=!_newfile:.m3u;=;!"
  set "_newfile=!_newfile:;=!.pla.refs"
)
if "!_direction!"=="p2m" (
rem from .pla.refs to .m3u
  rem if original filename has .pla.refs extension, strip it,
  rem and append .m3u; if not just append .m3u
  
  set "_newfile=!%~1!;"
  set "_newfile=!_newfile:.pla.refs;=;!"
  set "_newfile=!_newfile:;=!.m3u"
)

goto :eof

:STRIP_EXTENSION
:: removed


rem -----------------------------------
:CREATE_EMPTY_PLA
rem -----------------------------------
rem parameter - name of .pla.refs file
rem NB no check is made on this filename!
rem this only works correctly if it is a .pla.refs file
rem strip extension - get drive, path, name
rem file.pla.refs -> file.pla 
set "_emptypla=!%~1!"
set "_emptypla=!_emptypla:~0,-5!"
rem /y means overwrite if it exists
type NUL>"!_emptypla!"
goto :eof


rem -----------------------------------
:DELETE_EMPTY_PLA
rem -----------------------------------
rem parameter - name of .pla.refs file
rem NB no check is made on this filename!
rem this only works correctly if it is a .pla.refs file
rem strip extension - get drive, path, name
rem file.pla.refs -> file.pla 
set "_emptypla=!%~1!"
set "_emptypla=!_emptypla:~0,-5!"
if EXIST !_emptypla! (
  erase "!_emptypla!"
  if %ERRORLEVEL% NEQ 0 (
    echo WARNING: could not delete file !_emptypla!
  )
)
goto :eof

But as aGerman said:
Exclamation mark ('!') causes the same issue (i haven't corrected that).

penpen

[Quisquose]

Thank you, penpen.

In my complete ignorance when I started this thread, I really thought that the fix would just be a matter of adding few double quotation marks here and there, and perhaps also some kind of 'escape' character to prevent certain characters from being interpreted as commands. I had no idea that the problems with the batch file were so fundamental and pervasive.

I will look closely at the code that you have provided, to try to understand a bit more how you have addressed the issue. This will be easier for me to do seeing as you have provided details on exactly which lines should be replaced with what.

I tried to do a file diff on the amended version of the batch that Compo kindly supplied me with, but there were numerous line spacing differences in his version that essentially made it look as if every line in the entire file had been changed (even though it obviously hadn't). It was therefore difficult for me to see what he had done, although (as I said above) it has improved things considerably for me by essentially 'solving' the problem.

Your solution is helpful because files with ampersands in the name are output with changed filenames silently, so I may not even notice that it has happened until devices that looking for that particular file are unable to find it. However, exclamation marks generate a console error and halt the script, so that particular problem would not slip by me, because the error would immediately be brought to my attention and could therefore be addressed with temporary renaming.

[Compo]

@Quisquose, I have included the changes/replacements to the functions as mentioned above by penpen and re-attached it here as v0.5.

All of the other changes mentioned had already been implemented in the version I sent previously.

[Quisquose]

Many thanks, Compo.

[Quisquose]

Just an update, FYI.

There were a few errors in the v0.5 that was provided. They were mainly exclamation marks that were left in that penpen had specifically removed. There was also some differences in a couple of the new functions.

This resulted in converted files with % behaving even more strangely than the original. Instead of the % just being stripped from the output file name, one of the output files lost its filename altogether (it was named "~0,-5" with no extension) while the other output file had mostly the correct name but only one of the percentage signs that I included in the file name was retained in the output file.

I have since gone through line-by-line and made the replacements exactly as penpen provided, and it now works perfectly for files with percentage signs (in addition to continuing to work correctly with ampersands and parentheses as previously fixed by Compo).

Thanks again to both of you for your kind assistance.

[penpen]

It was surprisingly easy to handle exclamation marks correctly.
Replace these lines

Code: Select all

91:  if "%~3"=="" (
98:  set "_file=%~3"
187:  if "%~3"=="" (
194:  set "_inputdir=%~f3"

with the following lines:

Code: Select all

91:  if "!_param3.t3!"=="" (
98:  set "_file=!_param3.t3!"
187:  if "!_param3.t3!"=="" (
194:  set "_inputdir=!_param3.tf3!"

And then change the first lines of the batch file to:

Code: Select all

@echo off
rem playlist converter
rem v0.4 PSR
SETLOCAL ENABLEEXTENSIONS disableDelayedExpansion
set "_param3.t3=%~3"
set "_param3.tf3=%~f3"
SETLOCAL ENABLEDELAYEDEXPANSION

But you should note that therte are some other bugs in your batch file, for example:
- Within a compbound statement (aka block: "( commands )"), the following is nonsens:

Code: Select all

    set _inputdir2=!_inputdir!
    for /R %_inputdir2% %%F in (!_fileset!) do (

(%_inputdir2% is always empty.)
- I'm also pretty sure, that exclamation marks aren't handled correctly inside the playlist files.
- May unexpectedly fail with only some unicode named Files inside the playlists depending on your default codepage ("C:\foo\⚽.mp3") - should use utf-8 codepage for these if i remember right.


penpen

[Quisquose]

Penpen, thank you again for your helpful and insightful comments.

I have applied the latest fixes that you provided, and the converter batch now works flawlessly with playlists that contain exclamation marks in their names.

This script was created about 7 years ago, so I can't remember that much detail about the process, but I do remember specifically having a conversation with my friend about how he would write the script to handle instances where less common characters might appear in track names within the playlist.

Perhaps our focus was more on the track names inside the playlist because those names are obviously decided by the artist (and therefore they might contain all sorts of characters) whereas the name that given to one of my playlists is entirely up to me (and, as a native English speaker, I have no reason to ever use accented characters).

I recall doing some testing with a variety of different symbols and accented characters in track names within the playlist, and they all seemed to work fine. However, your astute observation and understanding of code has once again proved you correct, in that I have only just found out that you are right that exclamation marks are not handled correctly even within the playlist. This is something that I had not noticed until you mentioned it (despite me having used the converter script for years).

As flawed as the batch file is, it has still saved me immeasurable amounts of time over the years, and I'm therefore grateful to have it, because there is no way that I could have ever written something like this myself.

Thanks for drawing my attention to the potential for error even within the playlist; something which I was not really paying attention to up until you raised the issue. However, the likelihood of it being a real-world problem seems minimal, because otherwise I would have encountered the problem by now (given how long, and often, I have been using the converter script).