script to check urls

[olmos]

I am a beginner on scripting. Anyone know how I can script a way to do URL checking for a certain domain? I have a list of urls that I can also feed the script. I want it to be run automatically through a cron. Any examples or ideas?

Thanks in advance,
olmos

 

[feherke]

Hi

Some of the simplest variations with [tt]wget[/tt] :

[gray]# displays full HTTP response headers[/gray]

wget -S --spider -i /url/list > /output/file

[gray]# displays only brief result, but none if the host is unreachable[/gray]

wget -S --spider -nv -i /url/list > /output/file

[gray]# ensures each URL appears in the log[/gray]

while read u; do echo "$u"; wget -S --spider -nv "$u" ; done < /url/list > /output/file

[gray]# ensures each URL appears in the log and has status specified[/gray]

while read u; do echo "$u"; wget -S --spider -nv "$u" || echo "unreachable"; done < /url/list > /output/file

Feherke.

http://rootshell.be/~feherke/

 

[olmos]

Thanks for your help. One more thing, how do I check if a certain phrase is in each URL being checked?

Thanks again.
olmos

 

[feherke]

Hi

With [tt]grep[/tt] ? Something like this, for example :

[red]grep '[i]whatever[/i]' /url/list |[/red] wget -S --spider -i [red]-[/red] > /output/file

Feherke.

http://rootshell.be/~feherke/