So this is a little thing I find useful........ sorta............ sometimes....Robots.txt
Robots.txt are a way that admin can keep the sites directories all being pulled up by search engines. The way they can be useful is sometimes admin will list the directories specifically they don't want viewed, which could
show us the exact ones we were looking for to begin with. Where are these files saved? In the main directory. I.e. hackthissite.org/robots.txt
which prohibits /missions/ and /killing/all/humans/ from being viewed. So next time you are pentesting
a website take a quick look at the robots.txt, can't hurt to try.
p.s. I have yet to find out what this "/killing/all/humans/" is
if you have any info on that feel free to message me or reply here. *update* fas says it's a easter egg.
p.s.s. if this is in the wrong forum please move it, and my bad in advance.