how to get the size of chunk in http response using java if transfer encoding is chunked

Question

How to know the size of the chunk of HTTP response if transfer encoding is chunked.I am unable to get the logic. Please help me.And provide me some sample java code to get the size of chunk.I read in some books that size of each chunk is specified before the chunk itself.But using which logic can I get that. Please help me using java.

Thank you.

Supportable

Answer 1

Not able to give ready to use code but Apache HttpClient library supports "everything" out of the box.

This is wikipedia example of chunked response. You don't know the exact byte size of data until body part is read. Please note size of each chunk is hexadecimal value, last 0 sized chunk is end of data. It too is terminated by 2-byte CRLF delimiter.

Transfer-Encoding: chunked\r\n
Content-Type: text/plain\r\n
\r\n
4\r\n
Wiki\r\n
5;extkey=extvalue\r\n
pedia\r\n
E\r\n
.in\r\n
\r\n
chunks.\r\n
0\r\n
AfterBodyHeader: some value\r\n
AfterBodyHeader2: any value\r\n
\r\n

• read bytes until CRLF (\r\n or 0D0A in hex)
• drop everything after ; delimiter or stop at trailing CRLF, some replys may add extensions at the line of chunk size. Ignore them unless you wait for known key-value pair.
• convert hex to integer, it tells you num of data bytes in chunk
• read x num of bytes and append to ByteArrayOutputBuffer
• read trailing CRLF delimiter bytes and ignore it
• if not 0 sized read again size+chunk data
• some responses may add AfterHeaders they are like headers at the start of response
• after 0 sized chunk an empty line (\r\n) indicates end of complete http response. If there is no AfterHeaders you should see bytes 0\r\n\r\n in the end. Please note regular chunks may contain empty lines so do not terminate parsing until 0-sized indicator.

This example uses \r\n placeholders to indicate 2-byte delimiter as specs require

Hydrogeology

thank you..but I found in one page that chunk_size = int(line.strip().split(';')[0], 16) I did not understand this line.They have given this logic to find chunk size.but what is [0] index and why 16 is placed.I did not understand this statement.please can u help me. – Supportable

"5;extkey=extvalue".split(;) gives an array of two elements [0]="5", [1]="extkey=extvalue". int(arr[0],16) converts first string element to integer value, please note size of chunk is hexadecimal thats why an indicator of 16. If line does not have extension(;) split just returns one-sized array and strToInteger conversion still works. – Hydrogeology

the above logic i.e., int(arr[0],16) directly gives me the chunk size in bytes it seems.Am I correct?? – Supportable

int(arr[0],16) function converts hex string to integer value. Its the num of data bytes belonging to a next chunk. Yes 4 means "Wiki" bytes and 5 means "pedia" bytes. Please note \r\n terminator bytes at the end of each line and they are not included in chunk size. You just have to know there is those "hidden" two bytes on each line end. Size of "E" hex is 14 bytes ".in\r\n\r\nchunks." new lines within chunk data like any data bytes. – Hydrogeology

According to above example first I need to read 4\r\n and then convert 4 to integer. After conversion if it gives some value say X, then I have to read X bytes and append to ByteArrayOutputBuffer.that means this time I am reading "Wiki" right? and I should read \r\n which is there after "wiki" and ignore it.And again read 5\r\n and so on.Similar process continues until I get 0. Is this the way to get chunked data?Am I correct.Please give reply.thank you. – Supportable