How to check if a URL is online...

A place where newbies can post without (much) fear of reprisal. All mission posts should still go in the applicable forum.
Forum rules
Older HTS users: Be nice to the new people.

NEW USERS: This is NOT the place to post about missions! Refer to "Missions" category.

How to check if a URL is online...

Post by Zyphr on Fri Feb 17, 2012 2:25 pm
([msg=64436]see How to check if a URL is online...[/msg])

Hey guys,

I'm new here but plan to become a regular! I'll leave my introduction to the intro's section though.

My issue at hand is that I need to be able to check a long list of URL for a certain criteria. Ill show you what I need to do, what I've done so far, and what I really could do with some help with! Then you can either move to the next thread or put me on the right track :)

What I need to know

I need to know whether the site is currently online or offline, which is pretty simple. However I need it to be able to tell the difference between online and a holding page. Preferably I'd like it to class all sites with a holding page as offline also.

What I have so far

Well so far I have ready many different websites with many different implementations of code to do the online offline bit. The issue I'm having it the holding page bit. What I think needs to happen is to some how get the code to search the sites source code for certain keywords to determine whether it is a holding page or not.

The best code I came across was the below, however would be happy for your professional opinions for alternatives :)

Code: Select all
<?php

function Visit($url){

$agent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";$ch=curl_init();

curl_setopt ($ch, CURLOPT_URL,$url );

curl_setopt($ch, CURLOPT_USERAGENT, $agent);

curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt ($ch,CURLOPT_VERBOSE,false);

curl_setopt($ch, CURLOPT_TIMEOUT, 5);

curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, FALSE);

curl_setopt($ch,CURLOPT_SSLVERSION,3);

curl_setopt($ch,CURLOPT_SSL_VERIFYHOST, FALSE);

$page=curl_exec($ch);

//echo curl_error($ch);

$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);

curl_close($ch);

if($httpcode>=200 && $httpcode<300) return true;

else return false;

}

if (Visit("http://www.google.com"))

echo "Website OK"."n";

else

echo "Website DOWN";

?>


If you know where i should be looking to solve my issues, i would be extremely grateful :)

Thanks in advance,

Zyphr
Last edited by Zyphr on Fri Feb 17, 2012 5:41 pm, edited 1 time in total.
Zyphr
New User
New User
 
Posts: 3
Joined: Fri Feb 17, 2012 2:10 pm
Blog: View Blog (0)


Re: How to check if a URL is online...

Post by ghost107 on Fri Feb 17, 2012 5:00 pm
([msg=64442]see Re: How to check if a URL is online...[/msg])

If you want to know if a site is online or ofline you could use fsockopen():
http://php.net/manual/en/function.fsockopen.php

I usually use sockets, plus HTTP /1.1.
Code: Select all
$fp = fsockopen("http://www.example.com", 80, $errno, $errstr, 5);
if(!$fp){
    echo "offline";
}else{
   echo"online";
    $out = "GET / HTTP/1.1\r\n";
    $out .= "Host: www.example.com\r\n";
    $out .= "Connection: Close\r\n\r\n";         //Connection: Open
    fwrite($fp, $out);
    //process file
   fclose($fp);
}


http://www.w3.org/Protocols/rfc2616/rfc2616.html

You can use ajax + php too.
ghost107
Poster
Poster
 
Posts: 132
Joined: Wed Jul 02, 2008 7:57 am
Blog: View Blog (0)


Re: How to check if a URL is online...

Post by Zyphr on Sat Feb 18, 2012 1:03 pm
([msg=64457]see Re: How to check if a URL is online...[/msg])

Thanks for your reply!

Annoyingly enough, I have been told I must do this in javascript. Now my PHP was limited, however javascript is non existent...

Any idea how to get javascript to check if a site is online or offline, and then if online run Regex to determine if it is a holding page or not?

Tad lost here...
Zyphr
New User
New User
 
Posts: 3
Joined: Fri Feb 17, 2012 2:10 pm
Blog: View Blog (0)


Re: How to check if a URL is online...

Post by ghost107 on Sat Feb 18, 2012 6:53 pm
([msg=64465]see Re: How to check if a URL is online...[/msg])

If you want to check status of an page try using ajax, for more info read about ajax:

Code: Select all
httpRequest.onreadystatechange = ProcessMessage; 
httpRequest.open('GET', "http://www.example.com"); 
httpRequest.send(); 

function ProcessMessage() { 
    if (httpRequest.readyState === 4  && httpRequest.status === 200 ) { 
          alert('Online'); 
           //process message httpRequest.responseText
    } else { 
        alert('Offline.'); 
    } 
}
ghost107
Poster
Poster
 
Posts: 132
Joined: Wed Jul 02, 2008 7:57 am
Blog: View Blog (0)


Re: How to check if a URL is online...

Post by tgoe on Mon Feb 20, 2012 2:34 am
([msg=64490]see Re: How to check if a URL is online...[/msg])

@ghost107
I think you just accidentally the Same Origin Policy.

---

Looks like you're trying to write a domain squatter bot. JavaScript is neither needed nor helpful here. But I suppose you could do it with node or seed or something. Any details on why js is a requirement?
User avatar
tgoe
Contributor
Contributor
 
Posts: 633
Joined: Sun Sep 28, 2008 2:33 pm
Location: q3dm7
Blog: View Blog (0)


Re: How to check if a URL is online...

Post by tremor77 on Mon Feb 20, 2012 2:18 pm
([msg=64502]see Re: How to check if a URL is online...[/msg])

This would probably be best suited for a compiled php program, rather than a webpage script. And you are going through the effort of writing PHP for an executable you may as well just do in Perl or Python.
Image
User avatar
tremor77
Contributor
Contributor
 
Posts: 860
Joined: Wed Mar 31, 2010 12:00 pm
Location: New York
Blog: View Blog (0)


Re: How to check if a URL is online...

Post by ghost107 on Tue Feb 21, 2012 6:27 pm
([msg=64536]see Re: How to check if a URL is online...[/msg])

tgoe wrote:@ghost107
I think you just accidentally the Same Origin Policy.

Sorry about that, I never used ajax outside the same domain(Thanks for reminding me).
ghost107
Poster
Poster
 
Posts: 132
Joined: Wed Jul 02, 2008 7:57 am
Blog: View Blog (0)



Return to NZone

Who is online

Users browsing this forum: No registered users and 0 guests