How can I append DOM elements as they stream in from the network?

Question

I have an element that I want to populate as a request's HTML streams in, instead of waiting for the complete response. This turned out to be incredibly difficult.

Things I've tried:

1. milestoneNode.insertAdjacentHTML('beforebegin', text)

Be lovely if this worked. Unfortunately, elements with quirky parsing wreck it — such as <p> and <table>. The resulting DOM can be charitably described as Dada.

2. Using a virtual DOM/DOM update management library

Google's incremental-dom seemed most promising, but its patch() operation always restarts from the beginning of the container node. Not sure how to "freeze" it in place.

This also has baggage from doing at least HTML tokenization in JavaScript, and some actual tree-building would have to happen, unless one serves well-formed XHTML5. (Nobody does.) Reimplementing a browser's HTML parser seems like a sign I've gone horribly wrong.

3. document.write()

I was desperate. Ironically, this ancient boogeyman has almost the behavior I need, sans the "throwing away the existing page" thing.

4. Appending to a string, then innerHTMLing it periodically

Defeats the point of streaming, since eventually the entire response gets held in memory. Also has repeated serialization overhead.

On the plus side, it actually works. But surely there's a better way?

Philia

" as a request's HTML streams in" How is stream currently sent and received? – Backwardation

@Backwardation as a bona-fide Service Worker stream, spat onto the page with postMessage() – Philia

Answer 1

Jake Archibald figured out a silly hack to get this behavior in browsers today. His example code says it better than I would:

// Create an iframe:
const iframe = document.createElement('iframe');

// Put it in the document (but hidden):
iframe.style.display = 'none';
document.body.appendChild(iframe);

// Wait for the iframe to be ready:
iframe.onload = () => {
  // Ignore further load events:
  iframe.onload = null;

  // Write a dummy tag:
  iframe.contentDocument.write('<streaming-element>');

  // Get a reference to that element:
  const streamingElement = iframe.contentDocument.querySelector('streaming-element');

  // Pull it out of the iframe & into the parent document:
  document.body.appendChild(streamingElement);

  // Write some more content - this should be done async:
  iframe.contentDocument.write('<p>Hello!</p>');

  // Keep writing content like above, and then when we're done:
  iframe.contentDocument.write('</streaming-element>');
  iframe.contentDocument.close();
};

// Initialise the iframe
iframe.src = '';

Although <p>Hello!</p> is written to the iframe, it appears in the parent document! This is because the parser maintains a stack of open elements, which newly created elements are inserted into. It doesn't matter that we moved <streaming-element>, it just works.

Philia

He recently talked about a trick that doesn't need an iframe but uses the little-known const doc = document.implementation.createHTMLDocument() to create a new document in JavaScript land, append it to the main document (document.body.append(doc.body.firstChild)), and then use the doc.write() method of the detached document to stream in HTML to the page: youtu.be/LLRig4s1_yA?t=19m47s – Leavetaking

Answer 2

To iterate the nodes from streaming you can use the DOMParser and a marker:

parseHTMLStream:

const START_CHUNK_SELECTOR = "S-C";
const START_CHUNK_COMMENT = `<!--${START_CHUNK_SELECTOR}-->`;
const decoder = new TextDecoder();
const parser = new DOMParser();

/**
 * Create a generator that extracts nodes from a stream of HTML.
 *
 * This is useful to work with the RPC response stream and
 * transform the HTML into a stream of nodes to use in the
 * diffing algorithm.
 */
export default async function* parseHTMLStream(
  streamReader: ReadableStreamDefaultReader<Uint8Array>,
  ignoreNodeTypes: Set<number> = new Set(),
  text = "",
): AsyncGenerator<Node> {
  const { done, value } = await streamReader.read();

  if (done) return;

  // Append the new chunk to the text with a marker.
  // This marker is necessary because without it, we
  // can't know where the new chunk starts and ends.
  text = `${text.replace(START_CHUNK_COMMENT, "")}${START_CHUNK_COMMENT}${decoder.decode(value)}`;

  // Find the start chunk node
  function startChunk() {
    return document
    .createTreeWalker(
      parser.parseFromString(text, "text/html"),
      128, /* NodeFilter.SHOW_COMMENT */
      {
          acceptNode:  (node) =>  node.textContent === START_CHUNK_SELECTOR 
            ? 1 /* NodeFilter.FILTER_ACCEPT */
            : 2 /* NodeFilter.FILTER_REJECT */
      }
    )
    .nextNode();
  }

  // Iterate over the chunk nodes
  for (
    let node = getNextNode(startChunk());
    node;
    node = getNextNode(node)
  ) {
    if(!ignoreNodeTypes.has(node.nodeType)) yield node;
  }

  // Continue reading the stream
  yield* await parseHTMLStream(streamReader, ignoreNodeTypes, text);
}

/**
 * Get the next node in the tree. It uses depth-first 
 * to work with the streamed HTML.
 */
export function getNextNode(
  node: Node | null,
  deeperDone?: Boolean,
): Node | null {
  if (!node) return null;
  if (node.childNodes.length && !deeperDone) return node.firstChild;
  return node.nextSibling ?? getNextNode(node.parentNode, true);
}

How to use it:

const reader = res.body.getReader();

for await (const node of parseHTMLStream(reader)) {
  // each node from streaming, you can append to body or whatever
  console.log(node); 
}

I got this code with an attempt to do the dom diff algorithm with streaming, but I haven't got it yet. If someone gets it, I would be very grateful if you could share it here.

Uncourtly

Answer 3

You can use fetch(), process Response.body which is a ReadableStream; TextDecoder()

let decoder = new TextDecoder();

function handleResponse(result) {
  element.innerHTML += decoder.decode(result.value);
  return result
}

fetch("/path/to/resource/")
.then(response => response.body.getReader())
.then(reader => {
  return reader.read().then(function process(result) {
    if (result.done) {
      console.log("stream done");
      return reader.closed;
    }
    return reader.read().then(handleResponse).then(process)
  })
  .then(function() {
    console.log("stream complete", element.innerHTML);
  })
  .catch(function(err) {
    console.log(err)
  })
});

Backwardation

Yep, got that part so far. I'm then appending it to an element without reloading the page. – Philia

If you are using ServiceWorker you can use .respondWith Response instead of postMessage, though processing Response from fetch should return same result – Backwardation

I'm trying to avoid the full-page navigation that comes with .respondWith. I know it's weird, but it's for science. – Philia

You can use fetch(), Response.body.getReader(), TextDecoder, see updated post – Backwardation

This mangles the DOM for quirky-parse elements kind of like Attempt #1, unfortunately, and reserializes the entire element each update with the painter's algorithm. It certainly is simpler than some of what I did, though. – Philia

@Philia Yes, that would be an issue; you could parse the html at handleResponse to only set, reset .innerHTML when n-bytes, or n-elements have been successfully parsed. Another approach could be to perform several streams; each with a specific chunk of elements to process. When the element or element container is rendered begin next stream; or process next part of single stream. – Backwardation

@Philia That is, if you are aware .length of buffer when comprises a complete element container, you could process that chunk before proceeding to next chunk of data containing next element. This would involve counting the exact number of characters which complete an elements .outerHTML before calling fetch to retrieve the resource, when n.length is reached which completes the elements .outerHTML, append the element to container – Backwardation

@Philia See also CountQueuingStrategy – Backwardation