Page 1 of 1

Web Spider

PostPosted: Thu Aug 12, 2010 5:20 pm
by Calian
Just wondering if any recommendations for a freeware/shareware webspider? have done a bit of searching, just wanted 2nd opinion.


Re: Web Spider

PostPosted: Sat Aug 14, 2010 10:46 am
by tgoe
This looks good: pavuk and wget might work too.
What's your ultimate goal though?

Re: Web Spider

PostPosted: Sat Aug 14, 2010 2:58 pm
by Calian
Ultimate goal is to learn to be honest, and from what I've been reading and seeing all over the place, gaining all the information I can before looking at an attack is where I should be starting.

Figured a web spider would be a good tool for pulling up info on any projects. If there a better way go ahead and tell me, just wanted something more customisable than google.

Thanks will take a look at what you've mentioned here.

Re: Web Spider

PostPosted: Sat Aug 14, 2010 3:13 pm
by Goatboy
You could use something like Paros or Webscarab.

Basically it's a proxy that runs on localhost, intercepting all of your web traffic. It can automatically follow links, building a virtual map of the website. It also does some basic vulnerability testing, though I haven't had much luck with it. A lot of false positives, too. Worth a shot though. Definitely beats following links manually. I'd recommend Paros.

Re: Web Spider

PostPosted: Sat Aug 14, 2010 3:59 pm
by Calian
Thanks for the new ideas, have a few machines set-up will drop them all down and have a tinker, see which I think will do what I need, now and in the long run.

Thanks again.