web mystery

[Youkomoi]

I founded nice web mystery game, I've been stuck for a long time in the same mission. now i asking help to others.
Many of friends are trying to solve this task a long time but no one has succeeded in. CAN YOU BEAT IT OR HACK?

Url to imbossiple mission: http://www.hectigo.net/arvoitus/009.php?solution=abba
Hint to the mission is: If you were a robot, what would you do now?

[not_essence2]

"Robot Exclusion Standard". That sounds like what the hint is getting at. That's my best guess, Google it as I'm not going to directly give the answer. Research, it should be easy to understand.