jQuery read AJAX stream incrementally?
Asked Answered
P

6

82

I have read this question but it doesn't exactly answer my question. Unfortunately, it looks like things have changed in in the XHR object since I last looked at AJAX, so it is no longer possible to directly access responseText before it is finished being populated.

I have to write a page that uses AJAX (preferably jQuery, but I am open to suggestions) to retrieve CSV data via HTTP from a server I have no control over. The response data could be quite large; a megabyte of text is not uncommon.

The server is stream-friendly. Is there still any way to get access to a stream of data as it is being returned, directly from JavaScript?

I do have the option of writing some PHP code that lives in the middle and uses some sort of "Comet" tech (long-polling, EventSource, etc), but I would prefer to avoid that if possible.

In case it is relevant, assume for this question that users have the latest version of Firefox/Chrome/Opera and old browser compatibility is not an issue.

Pelecypod answered 12/10, 2011 at 13:15 Comment(1)
I know this has been answered, I did something like this before, have a look, rip it off if you must jsfiddle.net/JmZCE/1Valuator
F
21

You're going to want to use straight up javascript for this. The reason is that you're going to want to continuously poll and not wait for the callbacks to fire. You don't need jQuery for this, it's pretty simple. They have some nice source code for this on the Ajax Patterns website.

Essentially, you'll just want to keep track of your last position in the response and periodically poll for more text past that location. The difference in your case is that you can subscribe to the complete event and stop your polling.

Fassett answered 12/10, 2011 at 13:31 Comment(6)
Can you point me to a working example? The link you gave says that "The responseText property of XMLHttpRequest always contains the content that's been flushed out of the server, even when the connection's still open." .. and, from what I've been reading, this is no longer the case in newer browsers.Pelecypod
Isn't that just in IE? I thought readyState 3 contains it in other browsers.Fassett
Primarily I was going by the NOTE in this jquery plugin: plugins.jquery.com/project/ajax-http-stream 'NOTE: It has come to my attention that this no longer works as of Firefox 3.0.11 (works in 3.0.8 on linux), IE8, or the latest version of Chrome. Apparently the trend is to disallow access to the xmlhttprequest.responseText before the request is complete (stupid imo). Sorry there's nothing I can do to fix this'Pelecypod
It turns out that this actually does work with straight-up javascript, after just trying it (at least with browsers that behave properly). Still hoping to find a jquery version so that it works correctly across all browsers, but for now this is the best answer after all.Pelecypod
dead links make me sadGreenaway
The link is from archive.org. Sometimes you have to click on those a few times before they load. It's gotta go way back for the files.Heaney
P
95

This is quite straightforward when outputting text or HTML. Below is an example.

(You'll run into issues if trying to output JSON however, which I'll tackle further down.)

PHP FILE

header('Content-type: text/html; charset=utf-8');
function output($val)
{
    echo $val;
    flush();
    ob_flush();
    usleep(500000);
}
output('Begin... (counting to 10)');
for( $i = 0 ; $i < 10 ; $i++ )
{
    output($i+1);
}
output('End...');

HTML FILE

<!DOCTYPE>
<html>
    <head>
        <title>Flushed ajax test</title>
        <meta charset="UTF-8" />
        <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js"></script>
    </head>
    <body>
        <script type="text/javascript">
        var last_response_len = false;
        $.ajax('./flushed-ajax.php', {
            xhrFields: {
                onprogress: function(e)
                {
                    var this_response, response = e.currentTarget.response;
                    if(last_response_len === false)
                    {
                        this_response = response;
                        last_response_len = response.length;
                    }
                    else
                    {
                        this_response = response.substring(last_response_len);
                        last_response_len = response.length;
                    }
                    console.log(this_response);
                }
            }
        })
        .done(function(data)
        {
            console.log('Complete response = ' + data);
        })
        .fail(function(data)
        {
            console.log('Error: ', data);
        });
        console.log('Request Sent');
        </script>
    </body>
</html>

What if I need to do this with JSON?

It's not actually possible to load a single JSON object incrementally (before it's fully loaded) because until you have the complete object, the syntax will always be invalid.

But if your response has multiple JSON objects, one after another, then it's possible to load one at a time, as they come down the pipe.

So I tweaked my code above by...

  1. Changing PHP FILE line 4 from echo $val; to echo '{"name":"'.$val.'"};'. This outputs a series of JSON objects.

  2. Changing HTML FILE line 24 from console.log(this_response); to

    this_response = JSON.parse(this_response);
    console.log(this_response.name);
    

    Note that this rudimentary code assumes that each "chunk" coming to the browser is a valid JSON object. This will not always be the case because you cannot predict how packets will arrive - you may need to split the string based on semi-colons (or come up with another separator character).

Don't use application/json

Do NOT For change your headers to application/json - I did this and it had me Googling for 3 days. When the response type is application/json, the browser waits until the response is complete, as in fully complete. The full response is then parsed to check if it is infact JSON. However our FULL response is {...};{...};{...}; which is NOT valid JSON. The jqXHR.done method assumes there was an error, because the complete response cannot be parsed as JSON.

As mentioned in the comments, you can disable this check on the client side by using:

$.ajax(..., {dataType: "text"})

Hope some people find this useful.

Pedantry answered 23/9, 2013 at 16:17 Comment(8)
Wow thank you sir, this was exactly what I was looking for! Very good example of how to use this technique with JSON.Until
Thanks a lot, this took me 1 minute to implement successfully. Great stuff.Ambulate
Invoke $.ajax with {dataType:"text"}, this will inhibit the intelligent guess (see api.jquery.com/jquery.ajax dataType)Seeder
I'm using your example to read JSON (in onprogress event) and everything work well on localhost, however on public IP I'm getting json syntax errors. I know it's weird. Do you have any idea why?Blende
Thanks for the comment. I've never used this code in production, so I cannot offer any decent advice. Sorry. The only thing I can say is (the standard stuff)... Check the console for errors. Put console.logs and debuggers in your code to find out exactly where it breaksPedantry
It makes sense that you cannot read a JSON object incrementally, because the syntax will always be invalid until it has loaded completely. But you should be able to read plain text using this method. If you really need JSON, you might need to include multiple JSON objects one at a time and parse that yourself - jQuery won't understand that though.Brigham
Yes, you can read JSON incrementally, using a streaming JSON parser such as oboe (oboejs.com). You do not need to change your JSON response to have multiple JSON objects, and it's probably better not to from a design perspectiveArmful
A note on the PHP: It's generally bad practice to create a JSON manually on the PHP end by concatenating strings (eg. echo '{"name":"'.$val.'"};'). Some better code might be echo json_encode(["name"=>$val]).";";.Arvell
L
34

Use XMLHttpRequest.js

https://github.com/ilinsky/xmlhttprequest

http://code.google.com/p/xmlhttprequest

  • Delivers unobtrusive standard-compliant (W3C) cross-browser implementation of the XMLHttpRequest 1.0 object
  • Fixes ALL browsers quirks observed in their native XMLHttpRequest object implementations
  • Enables transparent logging of XMLHttpRequest object activity

To use long polling with PHP:

output.php:

<?php
header('Content-type: application/octet-stream');

// Turn off output buffering
ini_set('output_buffering', 'off');
// Turn off PHP output compression
ini_set('zlib.output_compression', false);
// Implicitly flush the buffer(s)
ini_set('implicit_flush', true);
ob_implicit_flush(true);
// Clear, and turn off output buffering
while (ob_get_level() > 0) {
    // Get the curent level
    $level = ob_get_level();
    // End the buffering
    ob_end_clean();
    // If the current level has not changed, abort
    if (ob_get_level() == $level) break;
}
// Disable apache output buffering/compression
if (function_exists('apache_setenv')) {
    apache_setenv('no-gzip', '1');
    apache_setenv('dont-vary', '1');
}

// Count to 20, outputting each second
for ($i = 0;$i < 20; $i++) {
    echo $i.str_repeat(' ', 2048).PHP_EOL;
    flush();
    sleep(1);
}

run.php:

<script src="http://code.jquery.com/jquery-1.6.4.js"></script>
<script src="https://raw.github.com/ilinsky/xmlhttprequest/master/XMLHttpRequest.js"></script>

<script>
$(function() {
    var xhr = new XMLHttpRequest();
    xhr.open('GET', '/longpoll/', true);
    xhr.send(null);
    var timer;
    timer = window.setInterval(function() {
        if (xhr.readyState == XMLHttpRequest.DONE) {
            window.clearTimeout(timer);
            $('body').append('done <br />');
        }
        $('body').append('state: ' + xhr.readyState + '<br />');
        console.log(xhr.responseText);
        $('body').append('data: ' + xhr.responseText + '<br />');
    }, 1000);
});
</script>

This should output:

state: 3
data: 0
state: 3
data: 0 1
state: 3
data: 0 1 2
state: 3
data: 0 1 2 3
state: 3
data: 0 1 2 3 4
...
...
...
state: 3
data: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
state: 3
data: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
state: 3
data: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
done
state: 4
data: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 

For IE you need to look into XDomainRequest

http://blogs.msdn.com/b/ieinternals/archive/2010/04/06/comet-streaming-in-internet-explorer-with-xmlhttprequest-and-xdomainrequest.aspx

http://msdn.microsoft.com/en-us/library/cc288060(VS.85).aspx

Linked answered 2/11, 2011 at 2:30 Comment(9)
This doesn't seem to support readystate 3, not even in Chrome :(Pelecypod
@Josh, yes it does. But there are various quirks with long-polling. You need to send 2Kb of data before the read state will change, and also set the content type to application/octet-stream. See my updated post for a PHP example.Linked
I will see what I can do with this. It seems inevitable that I will need to have some PHP in the middle, since I can't control the content-type of the original response. But I also really would like to be able to support IE6/7 (unfortunately)...Pelecypod
+1 for showing all the obstacles that are on the way to unbuffered output :)Nonillion
what is a stable and fast way of making sure only new content is printed (client-side)? Since currently it keeps repeating all data over and over againSpatula
@Spatula pastebin.com/3Dbt2mhQ Depending on you needs though, you might need to implement a custom protocol. Such a read all data up until a ;.Linked
Why do i need to use echo $i.str_repeat(' ', 2048).PHP_EOL; and not just echo the $i ?Bartholomew
@Bartholomew because some browsers won't allow streaming until 2kb of output has been sent.Linked
Thank you so much. In my case, that output buffering subject was helpful.Subtilize
F
21

You're going to want to use straight up javascript for this. The reason is that you're going to want to continuously poll and not wait for the callbacks to fire. You don't need jQuery for this, it's pretty simple. They have some nice source code for this on the Ajax Patterns website.

Essentially, you'll just want to keep track of your last position in the response and periodically poll for more text past that location. The difference in your case is that you can subscribe to the complete event and stop your polling.

Fassett answered 12/10, 2011 at 13:31 Comment(6)
Can you point me to a working example? The link you gave says that "The responseText property of XMLHttpRequest always contains the content that's been flushed out of the server, even when the connection's still open." .. and, from what I've been reading, this is no longer the case in newer browsers.Pelecypod
Isn't that just in IE? I thought readyState 3 contains it in other browsers.Fassett
Primarily I was going by the NOTE in this jquery plugin: plugins.jquery.com/project/ajax-http-stream 'NOTE: It has come to my attention that this no longer works as of Firefox 3.0.11 (works in 3.0.8 on linux), IE8, or the latest version of Chrome. Apparently the trend is to disallow access to the xmlhttprequest.responseText before the request is complete (stupid imo). Sorry there's nothing I can do to fix this'Pelecypod
It turns out that this actually does work with straight-up javascript, after just trying it (at least with browsers that behave properly). Still hoping to find a jquery version so that it works correctly across all browsers, but for now this is the best answer after all.Pelecypod
dead links make me sadGreenaway
The link is from archive.org. Sometimes you have to click on those a few times before they load. It's gotta go way back for the files.Heaney
N
16

Since you say your server is stream friendly (asynchronous) and was looking for a jquery solution, have you checked out the jQuery Stream Plugin?

It is really easy to use and allows you to not really worry about much of anything. It has pretty good documentation as well.

Nad answered 1/11, 2011 at 12:12 Comment(7)
I can certainly take a look at this. On a quick skim of the API page, I don't see a way to send HTTP POST and Basic Authentication information to the server, but I'm sure it must be in there somewhere. Also maybe "stream friendly" was the wrong choice of term. I don't mean asynchronous or bi-directional. I meant that it sends back a large amount of data over time, in a stream, like a gigantic HTTP response. Also, meanwhile, I have found a non-jquery solution that should be "good enough" for my original purposes.Pelecypod
well for http post and basic authentication, you'd use straight jquery anyways.Nad
And how do I integrate "straight jquery anyways" with the jquery stream plugin? Docs are unclear on that point. Got an example?Pelecypod
you include jquery in your header, then you include the jquery stream plugin after that in your header. This process is the same for any jQuery plugin. To do an http post using jQuery, look at this documentation (api.jquery.com/jQuery.post or api.jquery.com/jQuery.ajax ).Nad
+1 It's turned into portal now and it looks really awesome, encompassing WebSockets and all. github.com/flowersinthesand/portalStrobile
@Strobile Portal has reached its End of Life and is no longer maintained! Use Vibe.Tyre
these libraries links are dead. moreover, you don't need a library, as you can just use the ~25 lines of code in the other answer.Armful
S
2

I had to supply a grid with a large JSON payload that kept running into the maximum allowed size limit. I was using MVC and jquery, and so I adapted the solution of AlexMorley-Finch above.

The server code was from "Streaming data using Web API". Also https://github.com/DblV/StreamingWebApi.

public class StreamingController : ApiController
{

    [HttpGet]
    [ActionName("GetGridDataStream")]
    public HttpResponseMessage GetGridDataStream(string id)
    {
        var response = Request.CreateResponse();
        DynamicData newData = new DynamicData();
        var res = newData.GetDataRows(id);
        response.Content = new PushStreamContent((stream, content, context) =>
        { 
            foreach (var record in res)
            {
                var serializer = new JsonSerializer();
                using (var writer = new StreamWriter(stream))
                {
                    serializer.Serialize(writer, record);
                    stream.Flush();
                }

               // Thread.Sleep(100);
            }

            stream.Close();
        });

        return response;
    }
}

This created a stream of {json object}{json object}{json object} that needed delimiting commas and surrounding [ ] to be parsed as json successfully.

The client code was supplied the missing characters thus:

 var jsonData = {}; 

 $.ajax("api/Streaming/GetGridDataStream/" + viewName, {
    xhrFields: {
            onprogress: function (e) { 
                // console.log(this_response);
            }
        }
    }, { dataType: "text" }) //<== this is important for JSON data
    .done(function (data) { 

        data = "[" + data.replace(/\}\{/gi, "},{") + "]";

        jsonData["DataList"] = JSON.parse(data);
        //more code follows to create grid
    })
    .fail(function (data) {
        console.log('Error: ', data);
    });

I hope this helps someone using .Net MVC and jQuery.

Sherrer answered 10/12, 2019 at 2:30 Comment(0)
A
0

Here is a straightforward way to achieve this using JQuery (as requested by the OP):

First, extend the ajax object to support onreadystatechange by running the below code from https://gist.github.com/chrishow/3023092 (appended at the bottom of this response). Then just call ajax using an onreadystatechange function that will check xhr.responseText for new text.

If you wanted to get even fancier, you could clear the responseText data each time you read it, such as described here).

For example, see https://jsfiddle.net/g1jmwcmw/1/, which will download the response from https://code.jquery.com/jquery-1.5.js and output it in chunks inside your console window, using the code below (which you can just copy into an html page and then open in your browser):

<!-- jquery >= 1.5. maybe earlier too but not sure -->
<script src=https://code.jquery.com/jquery-1.5.min.js></script>
<script>
/* One-time setup (run once before other code)
 *   adds onreadystatechange to $.ajax options
 *   from https://gist.github.com/chrishow/3023092)
 *   success etc will still fire if provided
 */
$.ajaxPrefilter(function( options, originalOptions, jqXHR ) {
    if ( options.onreadystatechange ) {
        var xhrFactory = options.xhr;
        options.xhr = function() {
            var xhr = xhrFactory.apply( this, arguments );
            function handler() {
                options.onreadystatechange( xhr, jqXHR );
            }
            if ( xhr.addEventListener ) {
                xhr.addEventListener( "readystatechange", handler, false );
            } else {
                setTimeout( function() {
                    var internal = xhr.onreadystatechange;
                    if ( internal ) {
                        xhr.onreadystatechange = function() {
                            handler();
                            internal.apply( this, arguments ); 
                        };
                    }
                }, 0 );
            }
            return xhr;
        };
    }
});

// ----- myReadyStateChange(): this will do my incremental processing -----
var last_start = 0; // using global var for over-simplified example
function myReadyStateChange(xhr /*, jqxhr */) {
    if(xhr.readyState >= 3 && xhr.responseText.length > last_start) {
        var chunk = xhr.responseText.slice(last_start);
        alert('Got chunk: ' + chunk);
        console.log('Got chunk: ', chunk);
        last_start += chunk.length;
    }
}

// ----- call my url and process response incrementally -----
last_start = 0;
$.ajax({
  url: "https://code.jquery.com/jquery-1.5.js", // whatever your target url is goes here
  onreadystatechange: myReadyStateChange
});

</script>
Armful answered 13/11, 2017 at 5:56 Comment(4)
OP here. The question was asked 6 years ago. Is this something that would have worked in 2011/2012? I'm no longer working on this project so I'm not going to be able to test your answer.Pelecypod
yes, it works fine with jquery 1.5 (jan 2011, code.jquery.com/jquery-1.5.min.js). As an example, you can just cut/paste the code above.Armful
I'll just have to trust you. I ran your exact code in multiple browsers and the entire response was in one "chunk" so it didn't really prove anything. Don't have time to fiddle around with it further.Pelecypod
You should be able to see it. I saved the above, verbatim, to a test.html file and opened it in Chrome, and the console window showed the response received in two chunks.Armful

© 2022 - 2024 — McMap. All rights reserved.