Page 1 of 1
Searching a pattern in a website using curl in cmd
Posted: 15 Apr 2020 10:09
by X3nion
Hey there!
I'd like to use curl (or, if it’s not possible, another tool) to search some patterns on a web site using a batch script. If the search is successful, it would be great if curl could somehow give back a positive sign (e.g. a "1" in case of success, and a "0" in case of no hits).
Does anyone have an idea how this could be done?
I'd be grateful for every reply!
Kind regards, X3nion
Re: Searching a pattern in a website using curl in cmd
Posted: 16 Apr 2020 01:32
by penpen
I('m not familiar with curl, but most external batch/bash commands are setting the errorlevel to 0 on successa (== if all went as planned), so i would try (in windows batch):
Code: Select all
curl whatever params you need
if %errorlevel% == 0 (
echo(Ok.
) else (
echo(Error.
)
penpen
Re: Searching a pattern in a website using curl in cmd
Posted: 16 Apr 2020 06:54
by X3nion
Hey penpen and thanks for your reply!
Is there maybe someone else familiar with Curl?
Kind regards,
X3nion
Re: Searching a pattern in a website using curl in cmd
Posted: 16 Apr 2020 13:05
by aGerman
Probably something like that:
Code: Select all
curl "https://www.foobar.com/" | >nul findstr /c:"search me" && echo found || echo not found
Note: curl outputs the HTML source text. If you have bad luck it doesn't contain the pattern you are looking for because content might get loaded by scripts dynamically.
Steffen
Re: Searching a pattern in a website using curl in cmd
Posted: 16 Apr 2020 13:50
by X3nion
Hey aGerman and Thanks for your reply!
Do you come from Germany? If yes, we could write in German
I‘m going to try it out and answer if it worked out.
Kind regards,
X3nion
Re: Searching a pattern in a website using curl in cmd
Posted: 16 Apr 2020 15:57
by X3nion
Now I tried it and it worked out well, and it shows "Found"!
However, the following notification appears: "FINDSTR: Line 106 is too long".
What could that mean?
And how could I search for more strings?
I repeated the command more times, but I'm sure there must be an easier way?
Kind regards,
X3nion
Re: Searching a pattern in a website using curl in cmd
Posted: 16 Apr 2020 18:01
by aGerman
Do you come from Germany? If yes, we could write in German
I'm from Germany as well as penpen. However, this is an American forum and our users are from all over the world. Thus, English is the common language here and we're following this rule.
However, the following notification appears: "FINDSTR: Line 106 is too long".
What could that mean?
As I already wrote, curl outputs the HTML source. And HTML is still valid if the whole source text is just a single long line. Same with embedded JavaScript where a code line is terminated with a semicolon. No line wrapping necessary. So, it may appear that source lines are quite long - too long for FINDSTR which processes the text line-wise.
And how could I search for more strings?
Depends on what your goal is. You can pass several /c:"whatever" arguments to a single call of FINDSTR. The result is that FINDSTR succeeds if at least one of them was found. Not sure if this meets your requirements though.
Steffen
Re: Searching a pattern in a website using curl in cmd
Posted: 17 Apr 2020 03:59
by X3nion
Hey Steffen, thanks for your reply!
I'm from Germany as well as penpen. However, this is an American forum and our users are from all over the world. Thus, English is the common language here and we're following this rule.
Okay I of course understand this guideline!
What could that mean?
As I already wrote, curl outputs the HTML source. And HTML is still valid if the whole source text is just a single long line. Same with embedded JavaScript where a code line is terminated with a semicolon. No line wrapping necessary. So, it may appear that source lines are quite long - too long for FINDSTR which processes the text line-wise.
Would it maybe help to let curl save the data to a file and then search for the patterns?
Depends on what your goal is. You can pass several /c:"whatever" arguments to a single call of FINDSTR. The result is that FINDSTR succeeds if at least one of them was found. Not sure if this meets your requirements though.
Yes, that will do nicely!
Best regards,
X3nion aka Christian
Steffen
[/quote]
Re: Searching a pattern in a website using curl in cmd
Posted: 17 Apr 2020 06:34
by aGerman
Would it maybe help to let curl save the data to a file and then search for the patterns?
You could do this anyway, at least in order to evaluate if the lines that cause the error contain even data where you could find your patterns.
So, if you're asking for the code - it's pretty straightforward:
Code: Select all
>"out.txt" curl "https://www.foobar.com/"
>nul findstr "out.txt" /c:"search me" && echo found || echo not found
Steffen
Re: Searching a pattern in a website using curl in cmd
Posted: 17 Apr 2020 08:48
by X3nion
Hey and thanks for your reply!
When I provide the following code,
I get the following error:
FINDSTR: /c:Kann nicht geöffnet werden
not found
So in English: FINDSTR: /c: Cannot be opened.
not found
What could be the reason for that?
Kind regards,
Christian
Re: Searching a pattern in a website using curl in cmd
Posted: 17 Apr 2020 11:02
by ShadowThief
I've never seen findstr called that way before, and
findstr /? shows
findstr search_string file_name.
Code: Select all
>"out.txt" curl "https://www.foobar.com/"
>nul findstr /c:"search me" out.txt && echo found || echo not found
Step two, of course, is to open out.txt with a text editor and make sure that the thing you're looking for actually exists in the output.
Re: Searching a pattern in a website using curl in cmd
Posted: 17 Apr 2020 13:56
by aGerman
Jeez
Happens if you type code straight into the browser. Sorry.
Steffen
Re: Searching a pattern in a website using curl in cmd
Posted: 17 Apr 2020 15:23
by X3nion
Hey together,
now everything worked out very well - thanks a lot for your help!
Kind regards,
X3nion