C# How can I check if a URL exists/is valid?

Question

I am making a simple program in visual c# 2005 that looks up a stock symbol on Yahoo! Finance, downloads the historical data, and then plots the price history for the specified ticker symbol.

I know the exact URL that I need to acquire the data, and if the user inputs an existing ticker symbol (or at least one with data on Yahoo! Finance) it works perfectly fine. However, I have a run-time error if the user makes up a ticker symbol, as the program tries to pull data from a non-existent web page.

I am using the WebClient class, and using the DownloadString function. I looked through all the other member functions of the WebClient class, but didn't see anything I could use to test a URL.

How can I do this?

Guinea

updated to show C# 2.0 (VS2005) usage – Evenfall

Answer 1

You could issue a "HEAD" request rather than a "GET"? So to test a URL without the cost of downloading the content:

// using MyClient from linked post
using(var client = new MyClient()) {
    client.HeadOnly = true;
    // fine, no content downloaded
    string s1 = client.DownloadString("http://google.com");
    // throws 404
    string s2 = client.DownloadString("http://google.com/silly");
}

You would try/catch around the DownloadString to check for errors; no error? It exists...

---

With C# 2.0 (VS2005):

private bool headOnly;
public bool HeadOnly {
    get {return headOnly;}
    set {headOnly = value;}
}

and

using(WebClient client = new MyClient())
{
    // code as before
}

Evenfall

FWIW - Not sure if that really solves the problem (other than perhaps different behavior client side) since you are simply changing the HTTP method. The response from the server will depend heavily on how the logic is coded and may not work well for a dynamic service like stock price. For static resources (e.g. images, files etc) HEAD usually works as advertised since it is baked into the server. Many programmers do not explicitly HEAD requests since the focus is normally on POST and GET. YMMV – Austronesian

Sorry for taking so long to pick an answer... I got sidetracked with school and work and kind of forgot about this post. As a sidenote, I couldn't quite get your solution to work because I'm using Visual Studio 2005 which doesn't have the 'var' type. I haven't worked on this project in months, but is there simple fix for that fact? Also when I did try to implement your solution, I remember that it got mad at me for trying to define the HeadOnly property with no code in the 'get' and 'set' definitions. Or maybe I was just doing something wrong. Thanks for the help though! – Guinea

What is MyClient ? – Dietetic

@Dietetic there is a link in the body, to here: #153951 – Evenfall

Answer 2

Here is another implementation of this solution:

using System.Net;

///
/// Checks the file exists or not.
///
/// The URL of the remote file.
/// True : If the file exits, False if file not exists
private bool RemoteFileExists(string url)
{
    try
    {
        //Creating the HttpWebRequest
        HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
        //Setting the Request method HEAD, you can also use GET too.
        request.Method = "HEAD";
        //Getting the Web Response.
        HttpWebResponse response = request.GetResponse() as HttpWebResponse;
        //Returns TRUE if the Status code == 200
        response.Close();
        return (response.StatusCode == HttpStatusCode.OK);
    }
    catch
    {
        //Any exception will returns false.
        return false;
    }
}

From: http://www.dotnetthoughts.net/2009/10/14/how-to-check-remote-file-exists-using-c/

Siccative

I'm using this code to check whether a bunch of images exist, and it is quite slow (couple seconds per URL). Does someone know if this is an issue with this code, or just a fact of life when making these kinds of calls? – Cockalorum

@Cockalorum One way you could speed up your code is to do the check in a Parallel.Foreach loop if you hadn't tried that yet. It made my url testing app MUCH faster. – Guidon

This stuff throws DisposedObject in return (response.StatusCode == HttpStatusCode.OK); wrap in using – Superstitious

There is an issue with above code. if you do response.Close(); then you can not check for response.StatusCode as it's close it will throw an exception. – Deonnadeonne

@Cockalorum any method much faster ? – Dietetic

“too many automatic redirections were attempted” error message System.Net.WebException. View #518681 request.AllowAutoRedirect = false; and response.StatusCode == HttpStatusCode.Redirect (statusCode >= 100 && statusCode < 400) //Good requests) – Dietetic

Answer 3

These solutions are pretty good, but they are forgetting that there may be other status codes than 200 OK. This is a solution that I've used on production environments for status monitoring and such.

If there is a url redirect or some other condition on the target page, the return will be true using this method. Also, GetResponse() will throw an exception and hence you will not get a StatusCode for it. You need to trap the exception and check for a ProtocolError.

Any 400 or 500 status code will return false. All others return true. This code is easily modified to suit your needs for specific status codes.

/// <summary>
/// This method will check a url to see that it does not return server or protocol errors
/// </summary>
/// <param name="url">The path to check</param>
/// <returns></returns>
public bool UrlIsValid(string url)
{
    try
    {
        HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
        request.Timeout = 5000; //set the timeout to 5 seconds to keep the user from waiting too long for the page to load
        request.Method = "HEAD"; //Get only the header information -- no need to download any content

        using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
        {
            int statusCode = (int)response.StatusCode;
            if (statusCode >= 100 && statusCode < 400) //Good requests
            {
                return true;
            }
            else if (statusCode >= 500 && statusCode <= 510) //Server Errors
            {
                //log.Warn(String.Format("The remote server has thrown an internal error. Url is not valid: {0}", url));
                Debug.WriteLine(String.Format("The remote server has thrown an internal error. Url is not valid: {0}", url));
                return false;
            }
        }
    }
    catch (WebException ex)
    {
        if (ex.Status == WebExceptionStatus.ProtocolError) //400 errors
        {
            return false;
        }
        else
        {
            log.Warn(String.Format("Unhandled status [{0}] returned for url: {1}", ex.Status, url), ex);
        }
    }
    catch (Exception ex)
    {
        log.Error(String.Format("Could not test url {0}.", url), ex);
    }
    return false;
}

Christopher

I would add that some status codes in the 3xx range will actually cause an error to be thrown e.g. 304 Not Modified in which case you should be handling that in your catch block – Joleen

Just experienced a pull-your-hair-out problem with this approach: HttpWebRequest doesn't like it if you don't .Close() the response object before you try to download anything else. Took hours to find that one! – Aborning

HttpWebResponse object should be enclosed in using block since it implements IDisposable which will also ensure closing the connection. This might cause problems as @jbeldock, has faced. – Aer

It is throwing 404 Not Founds on urls that work fine in a browser... ? – Kemerovo

@MichaelTranchida Web servers are notoriously known for 404 when you issue a method which isn't supported. In your case Head mayn't be supported on that resource though Get could be. It should have thrown 405 instead. – Karafuto

For me, this seems to throw an exception if the server generates a 500 response, rather than entering the code block designed to catch the 400 status code. – Hepplewhite

If "http://" is not added to a URL, this method returns false – Jungian

Answer 4

A lot of the answers are older than HttpClient (I think it was introduced in Visual Studio 2013) or without async/await functionality, so I decided to post my own solution:

private static async Task<bool> DoesUrlExists(String url)
{
    try
    {
        using (HttpClient client = new HttpClient())
        {
            //Do only Head request to avoid download full file
            var response = await client.SendAsync(new HttpRequestMessage(HttpMethod.Head, url));

            if (response.IsSuccessStatusCode) {
                //Url is available is we have a SuccessStatusCode
                return true;
            }
            return false;
        }                
    } catch {
            return false;
    }
}

I use HttpClient.SendAsync with HttpMethod.Head to make only a head request, and not downlaod the whole file. Like David and Marc already say there is not only http 200 for ok, so I use IsSuccessStatusCode to allow all Sucess Status codes.

Kahle

Thanks! HttpWebRequest has become obsolete a while ago. – Endocardial

Answer 5

If I understand your question correctly, you could use a small method like this to give you the results of your URL test:

WebRequest webRequest = WebRequest.Create(url);  
WebResponse webResponse;
try 
{
  webResponse = webRequest.GetResponse();
}
catch //If exception thrown then couldn't get response from address
{
  return 0;
} 
return 1;

You could wrap the above code in a method and use it to perform validation. I hope this answers the question you were asking.

Farmhand

Yes, perhaps you can refine the solution by differentiating between different cases (TCP connection failure - host refuses connection, 5xx - Something fatal happened, 404 - Resource not found etc). Have a look at the Status property of WebException ;) – Austronesian

Very good point David! That would give us more detailed feedback so that we could handle the error more astutely. – Farmhand

Thanks. My point is that there are several layers to this onion, each of which can throw a wrench into the works (.Net Framework, DNS Resolution, TCP Connectivity, target Web Server, target application etc). IMHO a good design should be able to discriminate between the different failure conditions to provide informative feedback and usable diagnostics. Lets also not forget the HTTP has status codes for a reason ;) – Austronesian

Answer 6

I have always found Exceptions are much slower to be handled.

Perhaps a less intensive way would yeild a better, faster, result?

public bool IsValidUri(Uri uri)
{

    using (HttpClient Client = new HttpClient())
    {

    HttpResponseMessage result = Client.GetAsync(uri).Result;
    HttpStatusCode StatusCode = result.StatusCode;

    switch (StatusCode)
    {

        case HttpStatusCode.Accepted:
            return true;
        case HttpStatusCode.OK:
            return true;
         default:
            return false;
        }
    }
}

Then just use:

IsValidUri(new Uri("http://www.google.com/censorship_algorithm"));

Moline

Why not use result.IsSuccessStatusCode rather than a switch? – Vivienne

Good use of the switch to manage multiple status codes when needed! – Endocardial

Answer 7

Try this (Make sure you use System.Net):

public bool checkWebsite(string URL) {
   try {
      WebClient wc = new WebClient();
      string HTMLSource = wc.DownloadString(URL);
      return true;
   }
   catch (Exception) {
      return false;
   }
}

When the checkWebsite() function gets called, it tries to get the source code of the URL passed into it. If it gets the source code, it returns true. If not, it returns false.

Code Example:

//The checkWebsite command will return true:
bool websiteExists = this.checkWebsite("https://www.google.com");

//The checkWebsite command will return false:
bool websiteExists = this.checkWebsite("https://www.thisisnotarealwebsite.com/fakepage.html");

Britnibrito

Answer 8

WebRequest request = WebRequest.Create("http://www.google.com");
try
{
     request.GetResponse();
}
catch //If exception thrown then couldn't get response from address
{
     MessageBox.Show("The URL is incorrect");`
}

Kegan

Please add some explanation to your answer. Code-only answers tend to be confusing and not helpful to future readers and can attract downvotes that way. – Immobile

Answer 9

This solution seems easy to follow:

public static bool isValidURL(string url) {
    WebRequest webRequest = WebRequest.Create(url);
    WebResponse webResponse;
    try
    {
        webResponse = webRequest.GetResponse();
    }
    catch //If exception thrown then couldn't get response from address
    {
        return false ;
    }
    return true ;
}

Dhow

don't forget to close webResponse, else response time will grow each time you call your method – Halcomb

Answer 10

Here is another option

public static bool UrlIsValid(string url)
{
    bool br = false;
    try {
        IPHostEntry ipHost = Dns.Resolve(url);
        br = true;
    }
    catch (SocketException se) {
        br = false;
    }
    return br;
}

Benefice

That might be useful for checking if a host exists. The question is obviously not worried about whether or not the host exists. It is concerned with handling a bad HTTP path given the host is known to exist and be fine. – Vaudevillian

Answer 11

A lot of other answers are using WebRequest which is now obsolete.

Here is a method that has minimal code and uses currently up-to-date classes and methods.

I have also tested the other most up-voted functions which can produce false positives. I tested with these URLs, which points to the Visual Studio Community Installer, found on this page.

//Valid URL
https://aka.ms/vs/17/release/vs_community.exe

//Invalid URL, redirects. Produces false positive on other methods.
https://aka.ms/vs/14/release/vs_community.exe

using System.Net;
using System.Net.Http;

//HttpClient is not meant to be created and disposed frequently.
//Declare it staticly in the class to be reused.
static HttpClient client = new HttpClient();

/// <summary>
/// Checks if a remote file at the <paramref name="url"/> exists, and if access is not restricted.
/// </summary>
/// <param name="url">URL to a remote file.</param>
/// <returns>True if the file at the <paramref name="url"/> is able to be downloaded, false if the file does not exist, or if the file is restricted.</returns>
public static bool IsRemoteFileAvailable(string url)
{
    //Checking if URI is well formed is optional
    Uri uri = new Uri(url);
    if (!uri.IsWellFormedOriginalString())
        return false;

    try
    {
        using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Head, uri))
        using (HttpResponseMessage response = client.Send(request))
        {
            return response.IsSuccessStatusCode  && response.Content.Headers.ContentLength > 0;
        }
    }
    catch
    {
        return false;
    }
}

Just note that this will not work with .NET Framework, as HttpClient.Send does not exist. To get it working on .NET Framework you will need to change client.Send(request) to client.SendAsync(request).Result.

Cochabamba

Answer 12

Web servers respond with a HTTP status code indicating the outcome of the request e.g. 200 (sometimes 202) means success, 404 - not found etc (see here). Assuming the server address part of the URL is correct and you are not getting a socket timeout, the exception is most likely telling you the HTTP status code was other than 200. I would suggest checking the class of the exception and seeing if the exception carries the HTTP status code.

IIRC - The call in question throws a WebException or a descendant. Check the class name to see which one and wrap the call in a try block to trap the condition.

Austronesian

Actually, anything in the 200-299 range means success, IIRC – Evenfall

Marc, you absolutely are correct. I intentionally avoided get into the "class of error" concept (e.g. 5xx, 4xx, 3xx, 2xx etc) since that opens a whole other can of worms. Even handling the standard codes (200, 302, 404, 500 etc) is much better than ignoring the codes completely. – Austronesian

Answer 13

i have a more simple way to determine weather a url is valid.

if (Uri.IsWellFormedUriString(uriString, UriKind.RelativeOrAbsolute))
{
   //...
}

Badger

No, this method doesn't check whether the url is really accessible. It even returns true when Uri.IsWellFormedUriString("192.168.1.421", ...), which use an obviously incorrect url – Contraception

Answer 14

Following on from the examples already given, I'd say, it's best practice to also wrap the response in a using like this

    public bool IsValidUrl(string url)
    {
         try
         {
             var request = WebRequest.Create(url);
             request.Timeout = 5000;
             request.Method = "HEAD";

             using (var response = (HttpWebResponse)request.GetResponse())
             {
                response.Close();
                return response.StatusCode == HttpStatusCode.OK;
            }
        }
        catch (Exception exception)
        { 
            return false;
        }
   }

Bannon