DosTips.com

Hello DOStips,
I am looking for some advice on how to manage wget.

We are currently downloading with wget from a list of urls, called urls.txt

The log shows downloads that succeeded and also failed
I made a working batch to list the fails:
findstr /i "ERROR:" c:\urlstrip\test.txt >c:\urlstrip\certificate_errors.txt

ERROR: cannot verify fas.org's certificate, issued by `/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2':
ERROR: cannot verify www.berghahnjournals.com's certificate, issued by `/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2':

Where I am struggling, is to make a new list of ONLY the domains that failed (not the extra commentary), in a file (failed_domains.txt) so:
fas.org
www.berghahnjournals.com

...hopefully we can then use the failed_domains.txt and create a new file, with only urls from the urls.txt file that were also on the failed_domains.txt list

https://fas.org/irp/agency/dod/ig020907.pdf
https://fas.org/irp/agency/dod/ig020907.pdf
https://www.berghahnjournals.com/downlo ... 480111.pdf

etc.

Any help, even on part of it would be really appreciated.

Simple FOR /F command. It is the 4th delimited token based on spaces but you will also need to use the apostrophe as a delimiter as well to remove everything after the 's.

Squashman,
Thank you for the tip - I have never really understood the FOR /F command, so your suggestion made me pursue it a bit more diligently (thank you).

So these are baby steps for me, but when I run it from the command line it performs pretty well
FOR /F "tokens=1 delims='" %G IN (errors.txt) DO @echo %G
ERROR: cannot verify fas.org
ERROR: cannot verify www.berghahnjournals.com

BUT when I make it for a batch:
FOR /F "tokens=1 delims='" %%G IN (errors.txt) DO @echo %%G>clean.txt
...then it ignores the first line:
ERROR: cannot verify www.berghahnjournals.com

I am so new to FOR /F that I do not know why the batch versus command line would give different results - any ideas?

I imagine I will be able to FART.exe away the "ERROR: cannot verify" just fine.

Aisha

aisha wrote: ↑

01 Mar 2018 12:17

I am so new to FOR /F that I do not know why the batch versus command line would give different results - any ideas?
Aisha

You have a different perception of new. Myself and a few other people have given you several answers using FOR /F over the past 17 months you have been a member on this forum.

Read this page on redirection and you will see why your code is not working.