Web Spider

[Calian]

Just wondering if any recommendations for a freeware/shareware webspider? have done a bit of searching, just wanted 2nd opinion.

Ty

[tgoe]

This looks good: http://metha-sys.org/. pavuk and wget might work too.
What's your ultimate goal though?

[Calian]

Ultimate goal is to learn to be honest, and from what I've been reading and seeing all over the place, gaining all the information I can before looking at an attack is where I should be starting.

Figured a web spider would be a good tool for pulling up info on any projects. If there a better way go ahead and tell me, just wanted something more customisable than google.

Thanks will take a look at what you've mentioned here.

[Goatboy]

You could use something like Paros or Webscarab.

Basically it's a proxy that runs on localhost, intercepting all of your web traffic. It can automatically follow links, building a virtual map of the website. It also does some basic vulnerability testing, though I haven't had much luck with it. A lot of false positives, too. Worth a shot though. Definitely beats following links manually. I'd recommend Paros.

[Calian]

Thanks for the new ideas, have a few machines set-up will drop them all down and have a tinker, see which I think will do what I need, now and in the long run.

Thanks again.