Searching a pattern in a website using curl in cmd

[X3nion]

Hey there!

I'd like to use curl (or, if it’s not possible, another tool) to search some patterns on a web site using a batch script. If the search is successful, it would be great if curl could somehow give back a positive sign (e.g. a "1" in case of success, and a "0" in case of no hits).

Does anyone have an idea how this could be done?

I'd be grateful for every reply!

Kind regards, X3nion

[penpen]

I('m not familiar with curl, but most external batch/bash commands are setting the errorlevel to 0 on successa (== if all went as planned), so i would try (in windows batch):

Code: Select all

curl whatever params you need
if %errorlevel% == 0 (
	echo(Ok.
) else (
	echo(Error.
)

penpen

[X3nion]

Hey penpen and thanks for your reply!

Is there maybe someone else familiar with Curl?

Kind regards,
X3nion

[aGerman]

Probably something like that:

Code: Select all

curl "https://www.foobar.com/" | >nul findstr /c:"search me" && echo found || echo not found

Note: curl outputs the HTML source text. If you have bad luck it doesn't contain the pattern you are looking for because content might get loaded by scripts dynamically.

Steffen

[X3nion]

Hey aGerman and Thanks for your reply!
Do you come from Germany? If yes, we could write in German
I‘m going to try it out and answer if it worked out.

Kind regards,
X3nion

[X3nion]

Now I tried it and it worked out well, and it shows "Found"!

However, the following notification appears: "FINDSTR: Line 106 is too long".
What could that mean?

And how could I search for more strings?
I repeated the command more times, but I'm sure there must be an easier way?

Kind regards,
X3nion

[aGerman]

Do you come from Germany? If yes, we could write in German

I'm from Germany as well as penpen. However, this is an American forum and our users are from all over the world. Thus, English is the common language here and we're following this rule.

However, the following notification appears: "FINDSTR: Line 106 is too long".
What could that mean?

As I already wrote, curl outputs the HTML source. And HTML is still valid if the whole source text is just a single long line. Same with embedded JavaScript where a code line is terminated with a semicolon. No line wrapping necessary. So, it may appear that source lines are quite long - too long for FINDSTR which processes the text line-wise.

And how could I search for more strings?

Depends on what your goal is. You can pass several /c:"whatever" arguments to a single call of FINDSTR. The result is that FINDSTR succeeds if at least one of them was found. Not sure if this meets your requirements though.

Steffen

[X3nion]

Hey Steffen, thanks for your reply!

I'm from Germany as well as penpen. However, this is an American forum and our users are from all over the world. Thus, English is the common language here and we're following this rule.

Okay I of course understand this guideline!

What could that mean?
As I already wrote, curl outputs the HTML source. And HTML is still valid if the whole source text is just a single long line. Same with embedded JavaScript where a code line is terminated with a semicolon. No line wrapping necessary. So, it may appear that source lines are quite long - too long for FINDSTR which processes the text line-wise.

Would it maybe help to let curl save the data to a file and then search for the patterns?

Depends on what your goal is. You can pass several /c:"whatever" arguments to a single call of FINDSTR. The result is that FINDSTR succeeds if at least one of them was found. Not sure if this meets your requirements though.

Yes, that will do nicely!

Best regards,
X3nion aka Christian


Steffen
[/quote]

[aGerman]

Would it maybe help to let curl save the data to a file and then search for the patterns?

You could do this anyway, at least in order to evaluate if the lines that cause the error contain even data where you could find your patterns.
So, if you're asking for the code - it's pretty straightforward:

Code: Select all

>"out.txt" curl "https://www.foobar.com/"
>nul findstr "out.txt" /c:"search me" && echo found || echo not found

Steffen

[X3nion]

Hey and thanks for your reply!

When I provide the following code,

>"out.txt" curl "https://www.foobar.com/"
>nul findstr "out.txt" /c:"search me" && echo found || echo not found

I get the following error:

FINDSTR: /c:Kann nicht geöffnet werden
not found

So in English: FINDSTR: /c: Cannot be opened.
not found

What could be the reason for that?

Kind regards,
Christian

[ShadowThief]

I've never seen findstr called that way before, and findstr /? shows findstr search_string file_name.

Code: Select all

>"out.txt" curl "https://www.foobar.com/"
>nul findstr /c:"search me" out.txt && echo found || echo not found

Step two, of course, is to open out.txt with a text editor and make sure that the thing you're looking for actually exists in the output.

[aGerman]

Jeez Happens if you type code straight into the browser. Sorry.

Steffen

[X3nion]

Hey together,
now everything worked out very well - thanks a lot for your help!

Kind regards,
X3nion