How to use a proxy with in HtmlAgilityPack

Question

I need to use a proxy with HtmlAgilityPack. I give a link to my app RefURL. After that I want the app get url from a proxy address. For instance "101.109.44.157:8080"

I searched and found out this:

WebClient wc = new WebClient();
wc.Proxy = new WebProxy(host,port);
var page = wc.DownloadString(url);

and used it like this.

RefURL = new Uri(refLink.Text);

WebClient wc = new WebClient();
wc.Proxy = new WebProxy("101.109.44.157:8080");
var page = wc.DownloadString(RefURL);

RefURL.ToString();
HtmlWeb web = new HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = web.Load(RefURL.ToString());

but it does not work!

Angelitaangell

it does not work! is no sufficient description. – Sandblast

It cant serve the link via the proxy – Angelitaangell

The proxy is not responding. You need fresh proxy IP list. – Chenoweth

Answer 1

Need to pass the web proxy in this code line:

HtmlAgilityPack.HtmlDocument doc = web.Load(RefURL.ToString(),"GET", webProxy);

First step is finding "fresh proxy IP" list, for example:

• https://geonode.com/free-proxy-list/
• https://free-proxy-list.net/uk-proxy.html
• https://hidemy.name/en/proxy-list/
• https://proxyscrape.com/free-proxy-list
• http://free-proxy.cz

Most of these addresses would work for few hours. Check out how to set proxy IP in a browser. If the proxy is anonymous, this page (IP look up) should be unable to detect your actual location and IP.

Once you have a proxy IP and port that works, you can create webProxy object or simply pass IP and port.

string RefURL = "https://www.whatismyip.com/";
string myProxyIP = "119.81.197.124"; //check this is still available
int myPort = 3128;
string userId = string.Empty; //leave it blank
string password = string.Empty;
try
{
    HtmlWeb web = new HtmlWeb();
    var doc = web.Load(RefURL.ToString(), myProxyIP, myPort, userId, password);
    Console.WriteLine(doc.DocumentNode.InnerHtml);
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
}

Chenoweth

HtmlAgilityPack.HtmlDocument doc = web.Load(RefURL.ToString(),"ip", port,"",""); Thanks for your replay. I got this error An unhandled exception of type 'System.Net.WebException' occurred in System.dll Unable to connect to the remote server – Angelitaangell

Need new proxy server as explained in comments. These IPs are being blancklisted periodically and hence must be changed. – Chenoweth

I used a new proxy. it works fine with firefox or ie. – Angelitaangell