How to populate a large datalist (~2000 items) from a dictionary

Question

Right now I'm using the following code but it takes ~10 seconds on Chrome and ~2 minutes on IE11, which is where its mostly going to end up being used.

for (var key in dict) {
    if (dict.hasOwnProperty(key)) {
        combo.innerHTML += "<option value=\"" + dict[key] + "\">" + key + "</option>";
    }
}

I was reading this tutorial: http://blog.teamtreehouse.com/creating-autocomplete-dropdowns-datalist-element which suggested using ajax like so when dealing with larger quantities, though I'm not sure if large refers to 100 items or 100,000 items.

var request = new XMLHttpRequest();

request.onreadystatechange = function(response) {
  if (request.readyState === 4) {
    if (request.status === 200) {

      var jsonOptions = JSON.parse(request.responseText);

      jsonOptions.forEach(function(item) {

        var option = document.createElement('option');
        option.value = item;
        dataList.appendChild(option);

      });

    } else {
      console.log("Failed to load datalist options");
    }
  }
};

request.open('GET', 'html-elements.json', true);
request.send();

I've been trying to get this to work for a dictionary by replacing request.responseText with JSON.parse(JSON.stringify(dict)); but I'm running into problems getting it to make the request to begin with because it's not in a file.

How should I do this? And if I shouldn't be using a DataList for this, what alternative do you recommend?

Thanks in advance.

Morganne

Maybe #12604067 or tatiyants.com/… helps. It's another approach, they use a typeahead instead of a select box. – Finable

One major issue with your code is that you're touching the DOM ~2000 times. See https://mcmap.net/q/45226/-what-is-the-best-way-to-add-options-to-a-select-from-a-javascript-object-with-jquery for another solution which touches the DOM only once (with jQuery, but can be easily rewritten to vanilla JS) – Finable

If you have a large datalist, another option is dynamically inserting datalist items as you're typing. You can see this in the emoji selector on Chit Chat -- enter a colon (:), then search for an emoji; you can see the <datalist> being updated dynamically. – Sulfide

Answer 1

One area in which you could speed up performance is with a document fragment as writing to the DOM is slow.

var frag = document.createDocumentFragment();

for (var key in dict) {
    if (dict.hasOwnProperty(key)) {
        var option = document.createElement("OPTION");
        option.textContent = key;
        option.value = dict[key];
        frag.appendChild(option);
    }
}

combo.appendChild(frag);

Hakodate

Thanks, that's quite a bit faster. I tried appending to see if that improved the performance before, but I was appending an option to the combo each iteration, so it was equally bad. I've never used documentFragment before. Good to know. – Morganne

Answer 2

On instant way to get better performance is to build the HTML string up first, then assign it to the innerHTML.

var htmlStr = '';
for (var key in dict) {
    if (dict.hasOwnProperty(key)) {
        htmlStr += "<option value=\"" + dict[key] + "\">" + key + "</option>";
    }
}
combo.innerHTML = htmlStr;

The difference is huge: http://jsperf.com/string-append-vs-dom

Accessible

Answer 3

The DOM is notoriously slow. You could try filtering manually and only showing the first X elements. As znap026 pointed out, using document fragments will also help speed things up.

"use strict";

var data = Object.getOwnPropertyNames(window).sort(),
  datalist = document.getElementById("datalist"),
  input = document.getElementById("input");
  
const processedData = Object.fromEntries(data.map(d => [d.toLowerCase(), d]));

function search() {
  var term = input.value.toLowerCase();
  var found = 0;
  var frag = document.createDocumentFragment();

  for (var child of [].slice.apply(datalist.childNodes)) {
    datalist.removeChild(child);
  }

  for (var searchable in processedData) {
    if (searchable.indexOf(term) === 0) {
      let item = processedData[searchable];
      let option = document.createElement("option");
      option.value = item

      frag.appendChild(option);
      if (++found > 10) break;
    }
  }

  datalist.appendChild(frag);
}

search();
input.addEventListener("input", search);

<input id="input" list="datalist" placeholder="window properties"/>
<datalist id="datalist"></datalist>

Comeon

Very clever use of large datasets! – Mercuri

To best understand, if this is truly faster then it is also saying that JavaScript array filtering is faster then native C code (Chrome) doing the same filtering. I would have bet the farm that plopping the entire set into the datalist would allow Chrome to perform the same filtering but via optimized machine code instead of JIT JavaScript runtime code. As is above there are now two filter loops: one by the JS on the input change to mutate the DOM and the second by the browser also filtering the now filtered set. – Eolic

@Eolic - I believe it's slightly different than you describe. While the native C code may be faster than JavaScript at array filtering, it may not get to even start filtering until after a few thousand DOM nodes are created. Also, modern JavaScript engines are able to optimize certain "hot" functions... so even the difference in filtering performance may be negligible. I'd definitely recommend running some performance tests before betting that farm. All your chickens are belong to us! – Protuberance

Answer 4

You can also improve performance by detaching the datalist from then input and then attaching it again after populating it:

"use strict";

// init
const values = Object.getOwnPropertyNames(window);
const datalist = document.getElementById("datalist");
const arr = values.map(k => new Option(k));

let start = performance.now();
datalist.replaceChildren(...arr);
console.log('plain replace:', performance.now() - start);

// reset
datalist.replaceChildren(...[]);

start = performance.now();
datalist.setAttribute("id", undefined);
datalist.replaceChildren(...arr);
datalist.setAttribute("id", "datalist");
console.log('replace with detach and attach:', performance.now() - start);

<input list="datalist" />
<datalist id="datalist"></datalist>

Touristy