HttpConnection - javax.microedition, returning -1 for getLength() method

Question

I am trying to program a very simple Mobile Application (J2ME) in java. The idea is to access a website via URL input and read the contents of the website into a buffer.

Here's the problem. This works perfectly fine for some URL's but not others? The example below (wikipedia) works fine. But take "http://java.com/en/about/" as an example and the "HttpConnection hc" returns -1 for getLenght() so there is no content to read into the buffer?

Here's my code:

        String url = "http://en.wikipedia.org/wiki/RSS";

        //Sets up HttpConnection and InputStream using the URL variable
        HttpConnection hc = null;
        InputStream is = null;

        try {
            hc = (HttpConnection) Connector.open(url);
            is = hc.openInputStream();
        } catch (IOException ie) {
            System.out.println(ie.getMessage());
        }

        //Reader object created to read input from InputStream
        Reader rdr = new InputStreamReader(is);

        //Variable "content" will store HTML code
        String content = "";

        //Get the lenght of the data to set the buffer sizes
        int len = (int) hc.getLength();

Any ideas? let me know if I've missed anything out!

Just for info I am using Netbeans 6.9.1

Library for HttpConnection is "javax.microedition.io.HttpConnection;" + "import javax.microedition.io.Connector;"

Bannasch

Why don't you simply use a dynamically growing buffer such as a ByteArrayOutputStream (that should also be available in J2ME). – Duelist

This doesnt solve the problem of the hc.getLength being -1 though? – Bannasch

Answer 1

The HTTP response from java.com is

HTTP/1.1 200 OK
Server: Sun-Java-System-Web-Server/7.0
Date: Wed, 23 Feb 2011 11:07:44 GMT
Content-Type: text/html;charset=UTF-8
Set-Cookie: JSESSIONID=B62F3DFB233BB2806018EC721F6C3FD7; Path=/
Content-Encoding: gzip
Vary: accept-encoding
Transfer-Encoding: chunked

The HTTP response from wikipedia is

HTTP/1.0 200 OK
Date: Wed, 23 Feb 2011 10:18:56 GMT
Server: Apache
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
Content-Language: en
Vary: Accept-Encoding,Cookie
Last-Modified: Fri, 18 Feb 2011 00:23:59 GMT
Content-Encoding: gzip
Content-Length: 24905
Content-Type: text/html; charset=UTF-8
Age: 2984
X-Cache: HIT from sq61.wikimedia.org, MISS from sq38.wikimedia.org
X-Cache-Lookup: HIT from sq61.wikimedia.org:3128, MISS from sq38.wikimedia.org:80
Connection: keep-alive

As you see, the HTTP response of http://java.com/en/about/ doesn't contain Content-Length header, the content is chunked.

So, the getLength() return -1.

Stringendo

Thanks for the reply. Im guessing this is my problem then? "Transfer-Encoding: chunked" i need to perhaps create an if statement to work out if the content is "chunked" (so getLenght() is equal to -1) and if it is chunked then work out a way to but this "chunked" data into a buffer? – Bannasch

@FloweN, in fact, you don't need to check the length of the content. You just need to check the the end of the stream when read the content. – Stringendo

For example, final StringBuffer buf = new StringBuffer(); final BufferedReader br = new BufferedReader(new InputStreamReader(is)); String line; while ((line = br.readLine()) != null) { if (line.trim().length() > 0) { buf.append(line).append("\r\n"); } } br.close(); final String content = buf.toString(); – Stringendo

That makes sense! But i cant use BufferedReader in J2ME? – Bannasch

@FloweN, InputStreamReader is also ok. The read method of Reader will return -1 if the end of the stream has been reached. final StringBuffer buf = new StringBuffer(); final Reader reader = new InputStreamReader(is); final char[] c = new char[1024]; int n = 0; while (-1 != (n = reader.read(c))) { buf.append(c, 0, n); } reader.close(); final String content = buf.toString(); – Stringendo

Thanks so much for the help kliu , i understand how this works now :) – Bannasch