How to fetch HTML in Java

Question

Without the use of any external library, what is the simplest way to fetch a website's HTML content into a String?

Smithers

possible duplicate of #239047 – Slotter

Answer 1

I'm currently using this:

String content = null;
URLConnection connection = null;
try {
  connection =  new URL("http://www.google.com").openConnection();
  Scanner scanner = new Scanner(connection.getInputStream());
  scanner.useDelimiter("\\Z");
  content = scanner.next();
  scanner.close();
}catch ( Exception ex ) {
    ex.printStackTrace();
}
System.out.println(content);

But not sure if there's a better way.

Smithers

Why "\\Z"? Isn't it an EOF on Windows only? I am just guessing here. – Assessor

Why do you use "\\Z"? What does it do? I tried without it, it didn't work. – Waldo

@MaxHusiv I think it's because if you don't specify a delimiter, scanner.next() will just go through the whole HTML character by character, but if you use a delimiter which won't be found in the HTML, scanner.next() returns the whole thing. – Lodge

What import statements do you need for that to work? – Mease

Answer 2

This has worked well for me:

URL url = new URL(theURL);
InputStream is = url.openStream();
int ptr = 0;
StringBuffer buffer = new StringBuffer();
while ((ptr = is.read()) != -1) {
    buffer.append((char)ptr);
}

Not sure at to whether the other solution(s) provided are any more efficient or not.

Milliemillieme

Don't you need to include the following? import java.io.* import java.net.* – Dilks

Sure, but they're core java so very simple. As for the actual code, the import statements are omitted for clarity. – Milliemillieme

after while, you should display the buffer's content too! or write a method where you read it! – Raseda

be sure to close the inputstream – Brancusi

why have you named the variable ptr? – Shiah

Answer 3

I just left this post in your other thread, though what you have above might work as well. I don't think either would be any easier than the other. The Apache packages can be accessed by just using import org.apache.commons.HttpClient at the top of your code.

Edit: Forgot the link ;)

Contraception

Apparently you also have to install the JAR file :) – Dilks

Answer 4

Whilst not vanilla-Java, I'll offer up a simpler solution. Use Groovy ;-)

String siteContent = new URL("http://www.google.com").text

Milliemillieme

Answer 5

 try {
        URL u = new URL("https"+':'+'/'+'/'+"www.Samsung.com"+'/'+"in"+'/');
        URLConnection urlconnect = u.openConnection();
        InputStream stream = urlconnect.getInputStream();
        int i;
        while ((i = stream.read()) != -1) {
            System.out.print((char)i);
        }
    }
    catch (Exception e) {
        System.out.println(e);
    }

Incessant

Answer 6

Its not library but a tool named curl generally installed in most of the servers or you can easily install in ubuntu by

sudo apt install curl

Then fetch any html page and store it to your local file like an example

curl https://www.facebook.com/ > fb.html

You will get the home page html.You can run it in your browser as well.

Seiber

Squints eyes to show shock. This is a Java Question. – Venose