Java HTTP server sending chunked response
Asked Answered
A

3

7

I am working on a Java application which has a built in HTTP server, at the moment the server is implemented using ServerSocketChannel, it listens on port 1694 for requests:

        msvrCh = ServerSocketChannel.open();
        msvrCh.socket().bind(new InetSocketAddress(mintPort));
        msvrCh.configureBlocking(false);

A thread is installed to manage requests and responses:

        Thread thrd = new Thread(msgReceiver);
        thrd.setUncaughtExceptionHandler(exceptionHandler);
        thrd.start();

The thread is quite simple:

        Runnable msgReceiver = new Runnable() {
            @Override
            public void run() {
                try{
                    while( !Thread.interrupted() ) {
    //Sleep a short period between checks for new requests                          
                        try{
                            Thread.sleep(DELAY_BETWEEN_ACCEPTS);
                        } catch(Exception ex) {
                            ex.printStackTrace();
                        }                           
                        SocketChannel cliCh = msvrCh.accept();

                        if ( blnExit() == true ) {
                            break;
                        }                           
                        if ( cliCh == null ) {
                            continue;
                        }
                        processRequest(cliCh.socket());
                    }                       
                } catch (IOException ex) {
                    ex.printStackTrace();
                } finally {                     
                    logMsg(TERMINATING_THREAD + 
                            "for accepting cluster connections", true);

                    if ( msvrCh != null ) {
                        try {
                            msvrCh.close();
                        } catch (IOException ex) {
                            ex.printStackTrace();
                        }
                        msvrCh = null;
                    }
                }               
            }
        };

The main bulk of the code for dealing with the response is in the function processRequest:

private void processRequest(Socket sck) {
    try {
    //AJAX Parameters
        final String AJAX_ID            = "ajmid";
    //The 'Handler Key' used to decode response         
        final String HANDLER_KEY        = "hkey";
    //Message payload           
        final String PAYLOAD            = "payload";
    //Post input buffer size            
        final int REQUEST_BUFFER_SIZE   = 4096;
    //Double carriage return marks the end of the headers           
        final String CRLF               = "\r\n";

        BufferedReader in = new BufferedReader(new InputStreamReader(sck.getInputStream()));
        String strAMID = null, strHKey = null, strRequest;
        char[] chrBuffer = new char[REQUEST_BUFFER_SIZE];
        StringBuffer sbRequest = new StringBuffer();
        eMsgTypes eType = eMsgTypes.UNKNOWN;
        clsHTTPparameters objParams = null;
        int intPos, intCount;               
    //Extract the entire request, including headers         
        if ( (intCount = in.read(chrBuffer)) == 0 ) {
            throw new Exception("Cannot read request!");
        }
        sbRequest.append(chrBuffer, 0, intCount);           
        strRequest = sbRequest.toString();
    //What method is being used by this request?
        if ( strRequest.startsWith(HTTP_GET) ) {
    //The request should end with a HTTP marker, remove this before trying to interpret the data
            if ( strRequest.indexOf(HTTP_MARKER) != -1 ) {
                strRequest = strRequest.substring(0, strRequest.indexOf(HTTP_MARKER)).trim();
            }            
    //Look for a data marker
            if ( (intPos = strRequest.indexOf(HTTP_DATA_START)) >= 0 ) {
    //Data is present in the query, skip to the start of the data
                strRequest = strRequest.substring(intPos + 1);
            } else {
    //Remove the method indicator
                strRequest = strRequest.substring(HTTP_GET.length());                   
            }
        } else if ( strRequest.startsWith(HTTP_POST) ) {
    //Discard the headers and jump to the data
            if ( (intPos = strRequest.lastIndexOf(CRLF)) >= 0 ) {
                strRequest = strRequest.substring(intPos + CRLF.length());  
            }
        }
        if ( strRequest.length() > 1 ) {
    //Extract the parameters                    
            objParams = new clsHTTPparameters(strRequest);
        }            
        if ( strRequest.startsWith("/") == true ) {
    //Look for the document reference
            strRequest = strRequest.substring(1);               
            eType = eMsgTypes.SEND_DOC;             
        }
        if ( objParams != null ) {
    //Transfer the payload to the request
            String strPayload = objParams.getValue(PAYLOAD);

            if ( strPayload != null ) {
                byte[] arybytPayload = Base64.decodeBase64(strPayload.getBytes()); 
                strRequest = new String(arybytPayload);
                strAMID = objParams.getValue(AJAX_ID);
                strHKey = objParams.getValue(HANDLER_KEY);
            }
        } 
        if ( eType == eMsgTypes.UNKNOWN 
          && strRequest.startsWith("{") && strRequest.endsWith("}") ) {
    //The payload is JSON, is there a type parameter?
            String strType = strGetJSONItem(strRequest, JSON_LBL_TYPE);

            if ( strType != null && strType.length() > 0 ) {
    //Decode the type                   
                eType = eMsgTypes.valueOf(strType.toUpperCase().trim());
    //What system is the message from?
                String strIP = strGetJSONItem(strRequest, JSON_LBL_IP)
                      ,strMAC = strGetJSONItem(strRequest, JSON_LBL_MAC);                   
                if ( strIP != null && strIP.length() > 0
                 && strMAC != null && strMAC.length() > 0 ) {
    //Is this system known in the cluster?
                    clsIPmon objSystem = objAddSysToCluster(strIP, strMAC);

                    if ( objSystem != null ) {
    //Update the date/time stamp of the remote system                           
                        objSystem.touch();                          
                    }
    //This is an internal cluster message, no response required
                    return;
                }                   
            }
        }            
        String strContentType = null, strRespPayload = null;
        OutputStream out = sck.getOutputStream();
        byte[] arybytResponse = null;
        boolean blnShutdown = false;
        out.write("HTTP/1.0 200\n".getBytes());

        switch( eType ) {
        case SEND_DOC:
            if ( strRequest.length() <= 1 ) {
                strRequest = HTML_ROOT + DEFAULT_DOC;
            } else {
                strRequest = HTML_ROOT + strRequest;
            }
            logMsg("HTTP Request for: " + strRequest, true);

            if ( strRequest.toLowerCase().endsWith(".css") == true ) {
                strContentType = MIME_CSS;
            } else if ( strRequest.toLowerCase().endsWith(".gif") == true ) {
                strContentType = MIME_GIF;
            } else if ( strRequest.toLowerCase().endsWith(".jpg") == true ) {
                strContentType = MIME_JPG;
            } else if ( strRequest.toLowerCase().endsWith(".js") == true ) {
                strContentType = MIME_JS;
            } else if ( strRequest.toLowerCase().endsWith(".png") == true ) {
                strContentType = MIME_PNG;
            } else if ( strRequest.toLowerCase().endsWith(".html") == true 
                     || strRequest.toLowerCase().endsWith(".htm") == true ) {
                strContentType = MIME_HTML;
            }
            File objFile = new File(strRequest);

            if ( objFile.exists() == true ) {
                FileInputStream objFIS = new FileInputStream(objFile);

                if ( objFIS != null ) {
                    arybytResponse = new byte[(int)objFile.length()];

                    if ( objFIS.read(arybytResponse) == 0 ) {
                        arybytResponse = null;
                    }
                    objFIS.close();
                }
            }
            break;
        case CHANNEL_STS:
            strRespPayload = strChannelStatus(strRequest);
            strContentType = MIME_JSON;
            break;
        case CLUSTER_STS:
            strRespPayload = strClusterStatus();
            strContentType = MIME_JSON; 
            break;
        case MODULE_STS:
            strRespPayload = strModuleStatus(strRequest);
            strContentType = MIME_JSON;
            break;
        case NETWORK_INF:
            strRespPayload = strNetworkInfo(strRequest);
            strContentType = MIME_JSON;
            break;
        case NODE_STS:
            strRespPayload = strNodeStatus(strRequest);
            strContentType = MIME_JSON;
            break;
        case POLL_STS:
            strRespPayload = strPollStatus(strRequest);
            strContentType = MIME_JSON;
            break;
        case SYS_STS:
    //Issue system status               
            strRespPayload = strAppStatus();
            strContentType = MIME_JSON;
            break;          
        case SHUTDOWN:
    //Issue instruction to restart system
            strRespPayload = "Shutdown in progress!";
            strContentType = MIME_PLAIN;
    //Flag that shutdown has been requested             
            blnShutdown = true;
            break;
        default:
        }
        if ( strRespPayload != null ) {
    //Convert response string to byte array             
            arybytResponse = strRespPayload.getBytes();
    System.out.println("[ " + strRespPayload.length() + " ]: " + strRespPayload);           //HACK          
        }           
        if ( arybytResponse != null && arybytResponse.length > 0 ) {
            if ( strContentType == MIME_JSON ) {
                String strResponse = "{";

                if ( strAMID != null ) {
    //Include the request AJAX Message ID in the response
                    if ( strResponse.length() > 1 ) {
                        strResponse += ",";
                    }   
                    strResponse += "\"" + AJAX_ID + "\":" + strAMID;
                }
                if ( strHKey != null ) {
                    if ( strResponse.length() > 1 ) {
                        strResponse += ",";
                    }
                    strResponse += "\"" + HANDLER_KEY + "\":\"" + strHKey + "\"";
                }
                if ( strResponse.length() > 1 ) {
                    strResponse += ",";
                }
                strResponse += "\"payload\":" + new String(arybytResponse) 
                             + "}";                 
                arybytResponse = strResponse.getBytes();
            }
            String strHeaders = "";

            if ( strContentType != null ) {
                strHeaders += "Content-type: " + strContentType + "\n";                 
            }
            strHeaders += "Content-length: " + arybytResponse.length + "\n" 
                        + "Access-Control-Allow-Origin: *\n"
                        + "Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT\n"
                        + "Access-Control-Allow-Credentials: true\n"
                        + "Keep-Alive: timeout=2, max=100\n"
                        + "Cache-Control: no-cache\n" 
                        + "Pragma: no-cache\n\n";
            out.write(strHeaders.getBytes());
            out.write(arybytResponse);
            out.flush();                
        }
        out.close();
        sck.close();

        if ( blnShutdown == true ) {
            String strSystem =  mobjLocalIP.strGetIP();

            if ( strSystem.compareTo(mobjLocalIP.strGetIP()) != 0 ) {
    //Specified system is not the local system, issue message to remote system.
                broadcastMessage("{\"" + JSON_LBL_TYPE  + "\":\"" + 
                                                   eMsgTypes.SHUTDOWN + "\""
                               + ",\"" + JSON_LBL_TIME  + "\":\"" + 
                                           clsTimeMan.lngTimeNow() + "\"}");                            
            } else {
    //Shutdown addressed to local system                    
                if ( getOS().indexOf("linux") >= 0 ) {
    //TO DO!!!                  
                } else if ( getOS().indexOf("win") >= 0 ) {
                    Runtime runtime = Runtime.getRuntime();
                    runtime.exec("shutdown /r /c \"Shutdown request\" /t 0 /f");
                    System.exit(EXITCODE_REQUESTED_SHUTDOWN);
                }               
            }
        }
    } catch (Exception ex) {            
    } finally {
        if (sck != null) {
            try {
                sck.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
    }
}

I would like to implemented a chunked response, at present chunked responses are not supported by the code above.

[Edit] I've tried to implement a chunked response by adding the method:

    /**
     * @param strData - The data to split into chunks
     * @return A string array containing the chunks
     */
 public static String[] arystrChunkData(String strData) {
    int intChunks = (strData.length() / CHUNK_THRESHOLD_BYTESIZE) + 1;
    String[] arystrChunks = new String[intChunks];
    int intLength = strData.length(), intPos = 0;

    for( int c=0; c<arystrChunks.length; c++ ) {            
        if ( intPos < intLength ) {
    //Extract a chunk from the data         
            int intEnd = Math.min(intLength - 1, intPos + CHUNK_THRESHOLD_BYTESIZE);
            arystrChunks[c] = strData.substring(intPos, intEnd);
        }
    //Advance data position to next chunk           
        intPos += CHUNK_THRESHOLD_BYTESIZE;
    }       
    return arystrChunks;
}

The modified processRequest now looks like this:

        private void processRequest(Socket sck) {
    try {
        //AJAX Parameters
        final String AJAX_ID            = "ajmid";
        //The 'Handler Key' used to decode response         
        final String HANDLER_KEY        = "hkey";
        //Message payload           
        final String PAYLOAD            = "payload";
        //Post input buffer size            
        final int REQUEST_BUFFER_SIZE   = 4096;
        //Double carriage return marks the end of the headers           
        final String CRLF               = "\r\n";

        BufferedReader in = new BufferedReader(new InputStreamReader(sck.getInputStream()));
        String strAMID = null, strHKey = null, strRequest;
        char[] chrBuffer = new char[REQUEST_BUFFER_SIZE];
        StringBuffer sbRequest = new StringBuffer();
        eMsgTypes eType = eMsgTypes.UNKNOWN;
        clsHTTPparameters objParams = null;
        int intPos, intCount;               
        //Extract the entire request, including headers         
        if ( (intCount = in.read(chrBuffer)) == 0 ) {
            throw new Exception("Cannot read request!");
        }
        sbRequest.append(chrBuffer, 0, intCount);           
        strRequest = sbRequest.toString();
        //What method is being used by this request?
        if ( strRequest.startsWith(HTTP_GET) ) {
        //The request should end with a HTTP marker, remove this before trying to interpret the data
            if ( strRequest.indexOf(HTTP_MARKER) != -1 ) {
                strRequest = strRequest.substring(0, strRequest.indexOf(HTTP_MARKER)).trim();
            }            
        //Look for a data marker
            if ( (intPos = strRequest.indexOf(HTTP_DATA_START)) >= 0 ) {
        //Data is present in the query, skip to the start of the data
                strRequest = strRequest.substring(intPos + 1);
            } else {
        //Remove the method indicator
                strRequest = strRequest.substring(HTTP_GET.length());                   
            }
        } else if ( strRequest.startsWith(HTTP_POST) ) {
        //Discard the headers and jump to the data
            if ( (intPos = strRequest.lastIndexOf(CRLF)) >= 0 ) {
                strRequest = strRequest.substring(intPos + CRLF.length());  
            }
        }
        if ( strRequest.length() > 1 ) {
        //Extract the parameters                    
            objParams = new clsHTTPparameters(strRequest);
        }            
        if ( strRequest.startsWith("/") == true ) {
        //Look for the document reference
            strRequest = strRequest.substring(1);               
            eType = eMsgTypes.SEND_DOC;             
        }
        if ( objParams != null ) {
        //Transfer the payload to the request
            String strPayload = objParams.getValue(PAYLOAD);

            if ( strPayload != null ) {
                byte[] arybytPayload = Base64.decodeBase64(strPayload.getBytes()); 
                strRequest = new String(arybytPayload);
                strAMID = objParams.getValue(AJAX_ID);
                strHKey = objParams.getValue(HANDLER_KEY);
            }
        } 
        if ( eType == eMsgTypes.UNKNOWN 
          && strRequest.startsWith("{") && strRequest.endsWith("}") ) {
        //The payload is JSON, is there a type parameter?
            String strType = strGetJSONItem(strRequest, JSON_LBL_TYPE);

              if ( strType != null && strType.length() > 0 ) {
        //Decode the type                   
                eType = eMsgTypes.valueOf(strType.toUpperCase().trim());
        //What system is the message from?
                String strIP = strGetJSONItem(strRequest, JSON_LBL_IP)
                      ,strMAC = strGetJSONItem(strRequest, JSON_LBL_MAC);                   
                if ( strIP != null && strIP.length() > 0
                 && strMAC != null && strMAC.length() > 0 ) {
        //Is this system known in the cluster?
                    clsIPmon objSystem = objAddSysToCluster(strIP, strMAC);

                    if ( objSystem != null ) {
        //Update the date/time stamp of the remote system                           
                        objSystem.touch();                          
                    }
        //This is an internal cluster message, no response required
                    return;
                }                   
            }
        }            
        String strContentType = null, strRespPayload = null;            
        OutputStream out = sck.getOutputStream();
        byte[] arybytResponse = null;
        boolean blnShutdown = false;
        //Start the writing the headers
        String strHeaders = "HTTP/1.0 200\n"
                          + "Date: " + (new Date()).toString() + "\n"
                          + "Access-Control-Allow-Origin: *\n"
                          + "Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT\n"
                          + "Access-Control-Allow-Credentials: true\n"
                          + "Keep-Alive: timeout=2, max=100\n"
                          + "Cache-Control: no-cache\n" 
                          + "Pragma: no-cache\n";            
        out.write(strHeaders.getBytes());
        strHeaders = "";

        switch( eType ) {
        case SEND_DOC:
            if ( strRequest.length() <= 1 ) {
                strRequest = HTML_ROOT + DEFAULT_DOC;
            } else {
                strRequest = HTML_ROOT + strRequest;
            }
            logMsg("HTTP Request for: " + strRequest, true);

            if ( strRequest.toLowerCase().endsWith(".css") == true ) {
                strContentType = MIME_CSS;
            } else if ( strRequest.toLowerCase().endsWith(".gif") == true ) {
                strContentType = MIME_GIF;
            } else if ( strRequest.toLowerCase().endsWith(".jpg") == true ) {
                strContentType = MIME_JPG;
            } else if ( strRequest.toLowerCase().endsWith(".js") == true ) {
                strContentType = MIME_JS;
            } else if ( strRequest.toLowerCase().endsWith(".png") == true ) {
                strContentType = MIME_PNG;
            } else if ( strRequest.toLowerCase().endsWith(".html") == true 
                     || strRequest.toLowerCase().endsWith(".htm") == true ) {
                strContentType = MIME_HTML;
            }
            File objFile = new File(strRequest);

            if ( objFile.exists() == true ) {
                FileInputStream objFIS = new FileInputStream(objFile);

                if ( objFIS != null ) {
                    arybytResponse = new byte[(int)objFile.length()];

                    if ( objFIS.read(arybytResponse) == 0 ) {
                        arybytResponse = null;
                    }
                    objFIS.close();
                }
            }
            break;
        case CHANNEL_STS:
            strRespPayload = strChannelStatus(strRequest);
            strContentType = MIME_JSON;
            break;
        case CLUSTER_STS:
            strRespPayload = strClusterStatus();
            strContentType = MIME_JSON; 
            break;
        case MODULE_STS:
            strRespPayload = strModuleStatus(strRequest);
            strContentType = MIME_JSON;
            break;
        case NETWORK_INF:
            strRespPayload = strNetworkInfo(strRequest);
            strContentType = MIME_JSON;
            break;
        case NODE_STS:
            strRespPayload = strNodeStatus(strRequest);
            strContentType = MIME_JSON;
            break;
        case POLL_STS:
            strRespPayload = strPollStatus(strRequest);
            strContentType = MIME_JSON;
            break;
        case SYS_STS:
        //Issue system status               
            strRespPayload = strAppStatus();
            strContentType = MIME_JSON;
            break;          
        case SHUTDOWN:
        //Issue instruction to restart system
            strRespPayload = "Shutdown in progress!";
            strContentType = MIME_PLAIN;
        //Flag that shutdown has been requested             
            blnShutdown = true;
            break;
        default:
        }
        if ( strRespPayload != null ) {
        //Convert response string to byte array             
            arybytResponse = strRespPayload.getBytes();
        }           
        if ( arybytResponse != null && arybytResponse.length > 0 ) {
            boolean blnChunked = false;

            if ( strContentType != null ) {
                strHeaders += "Content-type: " + strContentType + "\n";                 
            }               
            if ( strContentType == MIME_JSON ) {
                String strResponse = "{";

                if ( strAMID != null ) {
        //Include the request AJAX Message ID in the response
                    if ( strResponse.length() > 1 ) {
                        strResponse += ",";
                    }   
                    strResponse += "\"" + AJAX_ID + "\":" + strAMID;
                }
                if ( strHKey != null ) {
                    if ( strResponse.length() > 1 ) {
                        strResponse += ",";
                    }
                    strResponse += "\"" + HANDLER_KEY + "\":\"" + strHKey + "\"";
                }
                if ( strResponse.length() > 1 ) {
                    strResponse += ",";
                }
                strResponse += "\"payload\":" + new String(arybytResponse) 
                             + "}";
        //How big is the response?
    if ( strResponse.length() > CHUNK_THRESHOLD_BYTESIZE ) {
                    blnChunked = true;
                    strHeaders += "Transfer-Encoding: chunked\n\n";
                    out.write(strHeaders.getBytes());
        //Slice up the string into chunks
                            String[] arystrChunks = arystrChunkData(strResponse);

                    for( int c=0; c<arystrChunks.length; c++ ) {
                        String strChunk = arystrChunks[c];

                        if ( strChunk != null ) {
                            String strLength = Integer.toHexString(strChunk.length()) + "\r\n";
                            strChunk += "\r\n";
                            out.write(strLength.getBytes());
                            out.write(strChunk.getBytes());
                        }                           
                    }
        //Last chunk is always 0 bytes                      
                    out.write("0\r\n\r\n".getBytes());
                } else {
                    arybytResponse = strResponse.getBytes();
                }
            }
            if ( blnChunked == false ) {    
                strHeaders += "Content-length: " + arybytResponse.length + "\n\n";                          
                out.write(strHeaders.getBytes());
                out.write(arybytResponse);
            }
            out.flush();                
        }
        out.close();
        sck.close();

        if ( blnShutdown == true ) {
            String strSystem =  mobjLocalIP.strGetIP();

            if ( strSystem.compareTo(mobjLocalIP.strGetIP()) != 0 ) {
        //Specified system is not the local system, issue message to remote system.
                broadcastMessage("{\"" + JSON_LBL_TYPE  + "\":\"" + 
                                                   eMsgTypes.SHUTDOWN + "\""
                               + ",\"" + JSON_LBL_TIME  + "\":\"" + 
                                           clsTimeMan.lngTimeNow() + "\"}");                            
            } else {
    //Shutdown addressed to local system                    
                if ( getOS().indexOf("linux") >= 0 ) {
        //TO DO!!!                  
                } else if ( getOS().indexOf("win") >= 0 ) {
                    Runtime runtime = Runtime.getRuntime();
                    runtime.exec("shutdown /r /c \"Shutdown request\" /t 0 /f");
                    System.exit(EXITCODE_REQUESTED_SHUTDOWN);
                }               
            }
        }
    } catch (Exception ex) {            
    } finally {
        if (sck != null) {
            try {
                sck.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
    }
}

I've read several specifications for Chunked responses and as far as I can tell I am sending data in the correct format, however I don't receive anything in the browser.

I may have wrongly assume that the browser would correctly piece together the chunks into one, but I could be wrong. The client side handler looks like this:

      this.responseHandler = function() {
try {    
  if ( mobjHTTP == null
  || !(mobjHTTP.readyState == 4 && mobjHTTP.status == 200)
  || !(mstrResponseText = mobjHTTP.responseText)
  || mstrResponseText.length == 0 ) {
    //Not ready or no response to decode      
    return;
  }
    //Do something with the response
    } catch( ex ) {
  T.error("responseHandler:", ex);
}

};

This handler is set-up elsewhere in the object:

    mobjHTTP.onreadystatechange = this.responseHandler;
Acquah answered 27/10, 2015 at 16:26 Comment(7)
Why do you need a chunked response? TCP does it alreadyToughie
I'm creating a JSON response which can be quite large > 4K. Some browsers like IE will tolerate larger packets. I want to ensure the solution works for all by keeping the maximum packet size to 768 bytes per chunk.Acquah
I am pretty sure you won't have problems there in any popular browser, there is a limit on GET url length, but on response? Are you sure you have an issue?Toughie
The request method is not relevant, GET or POST, its the response data that needs to be chunked because there is a limit in TCP on the size of a packet and HTTP restricts it further.Acquah
In my response handler routine, I am seeing a response which tells me that the chunks are being reassembled, however the data is prefixed with the chunk size...not sure why, but it invalidates the data.Acquah
Looking at the response, the data actually contains everything, all the chunks including the chunk sizes and the terminating 0 with CRLF, not sure why...could it be down to the headers?Acquah
Don't use Thread.sleep(). Use a Selector to tell you when there is something to accept.Perreira
A
4

Solved, not sure why, but removing the header:

  Transfer-Encoding: chunked

And also the chunk lengths at the beginning of each chunk resolved the issue, I still write the data in 768 byte chunks. This works reliably and very well.

Not sure why I had to do this.

Final method to produce chunks from data string:

    public static String[] arystrChunkData(String strData) {
            int intChunks = (strData.length() / CHUNK_THRESHOLD_BYTESIZE) + 1;
            String[] arystrChunks = new String[intChunks];
            int intLength = strData.length(), intPos = 0;

            for( int c=0; c<arystrChunks.length; c++ ) {            
                if ( intPos < intLength ) {
    //Extract a chunk from the data         
                    int intEnd = Math.min(intLength, intPos + CHUNK_THRESHOLD_BYTESIZE);
                    arystrChunks[c] = strData.substring(intPos, intEnd);
                    intPos = intEnd;
                }
            }       
            return arystrChunks;
        }

Loop to write chunks, no lengths at the beginning and no 0 byte at the end of the chunks required:

    String[] arystrChunks = arystrChunkData(strResponse);
    for( String strChunk : arystrChunks ) {
            if ( strChunk != null ) {
                    out.write(strChunk.getBytes());
            }                           
    }
Acquah answered 28/10, 2015 at 12:13 Comment(1)
You are writing data to the output stream in chunks of 768 bytes. However, I think that there is no flushing after each chunk. By not sending Transfer-Encoding: chunked header and also not sending chunk length before each chunk; effectively, you are sending the complete data in one response when the output stream gets flushed in the end. Shouldn't you be flushing the output stream after writing each chunk?Sikang
T
2

As I commented already, there is NOT an official limit on HTTP response size. TCP does this work for you. However, you can always configure your web server to implement such a policy by setting Content-Length :: 32 bit Integer max size or 64 bit for modern browsers (see here).

Technically, you can have unlimited responses using Chunked Transfer as you state on your post. In theory, this is used to bypass the max Content-Length.

Most commonly and if there is such a requirement for a huge JSON file (at least some MBs in size), you can use some sort of pagination logic via sequential AJAX requests. In your case, you could split your big JSON data to chunks programmatically and send each one via another AJAX request. Then, let Javascript perform the merging task.

Typically, a JSON response of some MB in size will load successfully on any browser. I suggest you take a look on this article; it is 3-years old, but I guess things are even better nowadays.

In short, the above benchmark states that JSON of size less than 35 MB will probably load successfully on any modern desktop browser. This, however, may not be the case for mobile browsers. For instance, there are some reports for mobile safari limitations on >10MB Json files.

Toughie answered 29/10, 2015 at 8:32 Comment(0)
A
0

If you transfer user Transfer-Encoding=chunked then each chunk of data must be preceded by the size of the chunk.

See here for a good explanation: https://en.wikipedia.org/wiki/Chunked_transfer_encoding

Apprehend answered 21/3, 2022 at 11:14 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.