General Purpose Spider

Put your programming skills to the test in these challenges.

General Purpose Spider

Post by szech696 on Sun Nov 13, 2016 11:37 pm
([msg=93096]see General Purpose Spider[/msg])

Hello again everyone,
I needed a spider for another project, and I figured someone else could probably find some use in it.

* Multi-Threaded
* Uses a depth-first search
* Easy to Use
* Proxies
* Random User-Agent

To use:
Just inherit from widow.Widow, and override parse(response) for custom parsing.

Enjoy ;)
New User
New User
Posts: 8
Joined: Thu Nov 10, 2016 11:13 pm
Blog: View Blog (0)

Return to Programming

Who is online

Users browsing this forum: No registered users and 0 guests