Permanent Programming 11

[sanddbox]

I am encountering a specific problem with the program I'm building to do this mission. So far I've written the code to connect to the HTS main page and fire off my login info, then grab the source for the resulting page so I can see if it works or not. Problem is I'm getting "Invalid Referrer". What's going on here?

Because of HTS's crappy code, you have to supply an http referer with every request (I'm pretty sure it has to come from HTS). Otherwise, you'll get the "invalid referer".

EDIT: ninja'd by msbachman

[xor-logic]

I am encountering a specific problem with the program I'm building to do this mission. So far I've written the code to connect to the HTS main page and fire off my login info, then grab the source for the resulting page so I can see if it works or not. Problem is I'm getting "Invalid Referrer". What's going on here?

<br><br>HTS seems to care about the referrer. It's a field you can send over the other headers; you're apparently sending the cookie, so somewhere in there, include a line that says "Referer: [somewhere on hts]."<br><br>I gave a similar solution to prog. 12 just today, but if you're still stuck try it with a browser with wireshark running in the background. That's what I did to know what to send.

I am encountering a specific problem with the program I'm building to do this mission. So far I've written the code to connect to the HTS main page and fire off my login info, then grab the source for the resulting page so I can see if it works or not. Problem is I'm getting "Invalid Referrer". What's going on here?

<br><br>Because of HTS's crappy code, you have to supply an http referer with every request (I'm pretty sure it has to come from HTS). Otherwise, you'll get the "invalid referer".<br><br>EDIT: ninja'd by msbachman

Thank you both, I found the referrer as a method of the URLConnection object, set it to the main HTS page and it cleared it right up... I think. I log myself out before running the program. The source for the page I get back is the home page, not logged in. But if I click on any link, it takes me to that link, logged in. Strange. But I'm 80% sure it's working, and that should make a lot of the programming challenges easier. Thanks.

[sanddbox]

I am encountering a specific problem with the program I'm building to do this mission. So far I've written the code to connect to the HTS main page and fire off my login info, then grab the source for the resulting page so I can see if it works or not. Problem is I'm getting "Invalid Referrer". What's going on here?

<br><br>HTS seems to care about the referrer. It's a field you can send over the other headers; you're apparently sending the cookie, so somewhere in there, include a line that says "Referer: [somewhere on hts]."<br><br>I gave a similar solution to prog. 12 just today, but if you're still stuck try it with a browser with wireshark running in the background. That's what I did to know what to send.

I am encountering a specific problem with the program I'm building to do this mission. So far I've written the code to connect to the HTS main page and fire off my login info, then grab the source for the resulting page so I can see if it works or not. Problem is I'm getting "Invalid Referrer". What's going on here?

<br><br>Because of HTS's crappy code, you have to supply an http referer with every request (I'm pretty sure it has to come from HTS). Otherwise, you'll get the "invalid referer".<br><br>EDIT: ninja'd by msbachman

Thank you both, I found the referrer as a method of the URLConnection object, set it to the main HTS page and it cleared it right up... I think. I log myself out before running the program. The source for the page I get back is the home page, not logged in. But if I click on any link, it takes me to that link, logged in. Strange. But I'm 80% sure it's working, and that should make a lot of the programming challenges easier. Thanks.

I've got a hunch that if you send in the login credentials and then get the source of another page (say the basic 1 mission page), it will show you logged in.

[xor-logic]

Because of HTS's crappy code, you have to supply an http referer with every request (I'm pretty sure it has to come from HTS). Otherwise, you'll get the "invalid referer".

Not sure if that's sarcasm, but I assume it is. But hey, I genuinely did not know that for a program to connect to a website, you had to send a referrer too. This is the first time I've ever tried to do this, so now I know.

[sanddbox]

You shouldn't have to send a referer - like I said, it's HTS's code.

The http referer by definition is supposed to be optional.

(And no, it's not sarcasm. I'm actually serious.)

[xor-logic]

So I've just finished my program in Java, and it worked just great until I added in the last part to send the information back to the page. Now whenever I run it, I get an exception telling me I can't write output after I've read input.

Is there anyone well versed in Java that can help me? Please pm me if you can, as obviously I can't post my program source here (little bit of a spoiler).

[UKCrack]

This happening even with a try/catch statement?
Try looking up other examples online to do with http post in java

You shouldn't have to send a referer - like I said, it's HTS's code.

The http referer by definition is supposed to be optional.

(And no, it's not sarcasm. I'm actually serious.)

HTS requires a referer for a reason, look up security issues like CRLF, XSS etc It does add complication to your programming, but nothing to cause major issues

[msbachman]

look up security issues like CRLF, XSS etc

Things might be done differently in the UK, but around these parts you shouldn't assume that senior forum members are ignorant of topics like these.

He's posted some 1500 times in a comp. security forum, so I'd feel confident that passing mention of these things may have been made in a couple of viewed topics at least.

nothing to cause major issues

That depends who your clientele is. If you're catering to average joes, keeping a close watch on the referer field is probably fine. Not 1 in 1000 will likely even notice (or care all that much).

One the other hand, if users are security conscious enough to realize that referer information is potentially revealing sensitive data to the server said users are contacting, it might be more of a problem.

Pfc Manning (i.e. suspect in wikileaks Afghanistan leak) better hope to all that is holy that his IP isn't coupled with referer information to some kind of Wikileaks submission page on some site. If that would be the case (he used tor so at least he's somewhat conscious of the risks ran by his activities, so I think this to be unlikely.).

And since you took issue with this:

The http referer by definition is supposed to be optional.

I'd refer you to this.

All in all, I think w3.org is at least superficially credible. Lol.

-- Fri Jul 30, 2010 8:12 pm --

Another thing, I don't know a ton about this so I could be wrong: how do you figure CRLF is particularly applicable to the referer field?

[UKCrack]

My apologies for any offense

[hungryhobo14]

So, I can read from the web page to get all the info, but I am having trouble submitting my answer. is this near correct? I am programming in python, on ubuntu

print "the solved string is" + solved

values = {'solution':solved, 'submitbutton':'submit'}

solution = urllib.urlencode(values)

request = urllib2.Request("http://www.hackthissite.org/missions/prog/11", solution)
request.add_header('Referer', 'http://www.hackthissite.org/')

url = urlOpener.open(request)


when I read the returned url, it is just the normal webpage, like I did not submit anything at all. I am new to webpage interaction and programming so any help would be nice