Using dc.js on the clientside with crossfilter on the server

Question

I am working on creating interactive visualizations for large datasets. The datasets cannot be loaded in the browser due to their size. We're using crossfilter on a node server to load and filter the data serverside. I was wondering if I could somehow couple the server side crossfilter filters with dc.js charts.

I am using d3.js right now to create charts but want to use the various charts that are already there in dc.js. We are basically filtering the data on the serverside and then listening to changes on the chart on the clientside and performing appropriate filtering on the server and communicating via AJAX.

If someone could suggest a way to do incorporate a similar AJAX communication mechanism between the dc.js client and crossfilter server, it would be very helpful.

Employment

Answer 1

The coupling between dc.js and crossfilter is not very tight, and this should be possible.

There was a recent conversation about swapping out crossfilter, which is almost the same problem, on the dc.js mailing list:

https://groups.google.com/forum/#!topic/dc-js-user-group/fkRoFHuqg4k

The main functions that you'll need to marshall are group.all and dimension.filter. Sorry, I don't know of anyone accessing crossfilter remotely, but I know enough people doing almost this, that I have no doubt it can work.

Here is a related issue: https://github.com/dc-js/dc.js/issues/602

Gast

Answer 2

Curious to hear a little more about your experience here. My findings have been that client-server latency is one of the main problem that Crossfilter addresses. Interested to hear if you find that Crossfilter on the server actually buys you anything and if the interactivity is good enough to be useful with a reasonable latency delay. Seems like it might be faster to use a database.

That said, in answer to your question, in my experience dc.js doesn't require Crossfilter dimensions and groups. It just requires objects that work like Crossfilter dimensions and groups. So if you provide dc.js with objects that have .top() methods but those .top() methods actually hit the server using AJAX and return a data structure that looks like what Crossfilter returns, it should work. I've only done this with very basic charts, so you may have to implement other methods if you are using more advanced features. I believe you will want to be sure to make the method calls block until they return data.

Ucayali

The dataset I'm using is pretty big (million records) for the browser. I hit browser memory limits at around 100k records. So loading the entire data on the browser and working with that is very difficult. With crossfilter on the server am able to work with the scale of the data. – Employment

That's fair. Depending on your data, a million records might be workable. Depends on the size of the record, of course. – Ucayali