Robots.txt

General technological topics without their own forum go here

Robots.txt

Post by VvGHOSTvV on Fri Aug 09, 2013 10:35 am
([msg=76795]see Robots.txt[/msg])

So this is a little thing I find useful........ sorta............ sometimes....

Robots.txt

Robots.txt are a way that admin can keep the sites directories all being pulled up by search engines. The way they can be useful is sometimes admin will list the directories specifically they don't want viewed, which could show us the exact ones we were looking for to begin with. Where are these files saved? In the main directory. I.e. hackthissite.org/robots.txt which prohibits /missions/ and /killing/all/humans/ from being viewed. So next time you are pentesting a website take a quick look at the robots.txt, can't hurt to try.

p.s. I have yet to find out what this "/killing/all/humans/" is :?: if you have any info on that feel free to message me or reply here. *update* fas says it's a easter egg.

p.s.s. if this is in the wrong forum please move it, and my bad in advance.
MAY THE FORCE BE WITH YOU
User avatar
VvGHOSTvV
New User
New User
 
Posts: 33
Joined: Thu Jul 19, 2012 7:09 pm
Location: Oklahoma
Blog: View Blog (0)


Re: Robots.txt

Post by mShred on Fri Aug 09, 2013 9:09 pm
([msg=76798]see Re: Robots.txt[/msg])

lol yeah /killing/all/humans is just a little joke really.
But yeah, robots.txt formally restricts bots from crawling those directories. If you google /missions/ on HTS, it'll pop up saying "Robots.txt prohibits blah blah blah" or whatever. Of course, any crawler that we create can easily completely ignore robots.txt..
Image

For those about to rock.
User avatar
mShred
Administrator
Administrator
 
Posts: 1743
Joined: Tue Jun 22, 2010 4:22 pm
Blog: View Blog (2)


Re: Robots.txt

Post by pretentious on Sat Aug 10, 2013 1:19 am
([msg=76800]see Re: Robots.txt[/msg])

I Came to this same conclusion a while back. By puting directories in your robots.txt file, you're advertising the places you don't want people looking at and at the same time, it may or may not be completely ignored the the web crawler. Then again, security through obscurity is really the bare minimum and if it's all you're doing to protect your stuff, you're gonna have a bad time
Goatboy wrote:Oh, that's simple. All you need to do is dedicate many years of your life to studying security.

IF you feel like exchanging ASCII arrays, let me know ;)
pretentious wrote:Welcome to bat country
User avatar
pretentious
Contributor
Contributor
 
Posts: 667
Joined: Wed Mar 03, 2010 12:48 am
Blog: View Blog (0)



Return to General

Who is online

Users browsing this forum: No registered users and 0 guests