Replace crossfilter data, restore dimensions and groups

Question

I'm using dc.js to render a nice bubble chart of a dataset. Underlying dc.js is crossfilter.

I'd like to smoothly refresh my chart with new data from the server. This issue on Github makes it clear that it is possible to do this by:

1. deleting all the data from the crossfilter
2. adding in the new data
3. calling dc.redrawAll().

I've got this working but in order to delete all the data, you first have to clear all the filters (because crossfilter.remove only removes the records matching the current filter).

I'd like to 'remember' how my data was filtered before, so I can reconstruct the filter again once I've replaced all the data. I'm willing to get into the guts of the crossfilter code, but any pointers would be helpful.

Additionally: if anyone knows a way of updating crossfilter data based on a unique key, that'd be gold dust!

Glimpse

Are you only adding data? Then I'd say use crossfilter.add([data]). If you need to remove data as well, I don't have much to offer, but you might want to take a look at this issue and comment if it sounds like the right direction: github.com/square/crossfilter/issues/109 – Belloir

@EthanJewitt I'm not adding or removing data. I'm updating existing data, each row uniquely keyed. Or does add have the facility to update based on a key? – Glimpse

Gotcha - then you do need to remove the old data and add new data (no ability to update existing keys as far as I know). That capability doesn't really exist at the moment as you will have to remove all filters in order to remove your data. Personally, I think this is something that needs to be addressed in Crossfilter, and there is some discussion on that issue about how to address it, but I don't think anyone has had time to tackle it. – Belloir

Answer 1

Thanks to @londonrob's Answer all data is now removed from the index. Here is a more functional approach for the resetting of data.

// reset the filter for a dimension
function resetDimensionFilter (dimension) {
  dimension.filter(null);
}

// reset filters for all given dimensions, 
// remove all data from index and
// return empty index
function resetData(ndx, dimensions) {
  // Clear all filters from dimensions, because `ndx.remove` 
  // only removes records matching the current filters.
  dimensions.forEach(resetDimensionFilter);

  // Remove all data from the cross filter
  ndx.remove();
}

Uam

Wow! That's great news. Can you link to the github fix/issue/PR which addresses this? – Glimpse

Not sure what you mean. I just rewrote your code a bit to do just the resetting. – Uam

Oh. So what does 'thanks to londonrob's answer all data is now removed from the index' refer to? I'm confused... – Glimpse

you pointed out that each filter of all the dimensions must be reset as well. Not only the ones on the index itself. – Uam

Hey Rob, @line-o.. I'm trying to implement this in absence of a reproducible example. My version seems to re-apply the original filter only to the graph it was created by, not the other graphs that should actually adjust to it. See this jsfiddle. The page refreshes successfully after 3s with new data. If you select e.g. Mr A before the refresh, the corresponding pie doesn't filter out the 2013 slice after refreshing as it should, unless you manually deselect and reselect the filter (Mr A). Have I done something wrong? – Ruddock

Answer 2

Here's what I ended up hacking together. It works perfectly well, although I'm sure it's ludicrously inefficient because all the dimensions have to be created from scratch:

var _xfilter = crossfilter({x:1, y:2},{x:3, y:4}),
    _dimensions = [];

_dimensions.push(_xfilter.dimension(function(d) { return d.x; });

// Unfilters all the given dimensions, removes all data
// from xf and adds newData to xf.
var _xfilter_reset = function(xf, dimensions, newData) {
    var i;
    for (i = 0; i < dimensions.length; i++) {
        // Clear all filters from this dimension.
        // Necessary because xf.remove only removes records
        // matching the current filter.
        dimensions[i].filter(null);
    }
    xf.remove(); // Remove all data from the crossfilter
    xf.add(newData);
    return xf;
}

// Resets the global crossfilter object and reapplies all
// current dc.js chart filters.
var _refresh_data = function(data) {
    var i, j,
        chart, oldFilters,
        allCharts = dc.chartRegistry.list();

    _xfilter = _xfilter_reset(_xfilter, _dimensions, data);     

    // Reset all filters using dc.js
    for (i = 0; i < allCharts.length; i++) {
        chart = allCharts[i];
        oldFilters = chart.filters(); // Get current filters
        chart.filter(null); // Reset all filters on current chart
        for (j = 0; j < oldFilters.length; j++) {
            // Set all the oldFilters back onto the chart
            chart.filter(oldFilters[j]);
        }
    }
    dc.redrawAll();
}

Glimpse

I think this may be optimally efficient with what we have now - all of the groups and reductions have to be recalculated anyway. You might try rebuilding the filters while there is no data so that crossfilter isn't shuffling indexes and reductions for each .filter call. Or we can hope for the feature @Ethan mentions above. – Torrid