Loading multiple CSV in DC.js, adding a value, and concatenating the results into a single dataTable

Question

I have four CSVs with the same header information, each representing a quarterly result within a year.

Therefore for one result I can load it and display it into a dataTable simple via

d3.csv("data/first-quarter"), function(dataQ1){
    dataQ1.forEach(function(d){
        d.callTypes = d['Call Types'];
        d.callDesc  = d['Call Description'];
        d.callVol = d['Call Volume'];
        d.quarter   = 'Q1'; 
    });

    var facts = crossfilter(dataQ1);
    var timeDimension = facts.dimension(function(d){
       return d.quarter;
    });

    dataTable
      ... //data table attributes

    dc.renderAll();
});

However complications arise when I try to retrieve from multiple sources and append the results.

One approach I took was to place all the file path names into an array and iterate through a forEach, with a flag to show when it was the last iteration to render the table. But this failed with a "Too many recursion" error.

And the next was to nest as such

d3.csv(filesPathNames[0], function(dataQ1){
  d3.csv(filesPathNames[1], function(dataQ2){
    d3.csv(filesPathNames[2], function(dataQ3){
      d3.csv(filesPathNames[3], function(dataQ4){

But both of these methods seem to not work due to the fact that I can't simply add one CSV value to another. So I think where I'm having an issue is that I'm not sure how to concatenate dataQ1, dataQ2, dataQ3, and dataQ4 properly.

Is the only solution to manually append one to another with an added value of Q1, Q2, Q3, and Q4 as the time dimension?

Tenebrous

The queue library should help here. – Lustreware

Wonderful question ! – Altaf

Answer 1

Like Lars said, you can use the queue library. Here is an example of how this might work:

Step 1) Queue up your files:

<script type="text/javascript" src="http://d3js.org/queue.v1.min.js"></script>
var q = queue()
    .defer(d3.csv, "data/first-quarter")
    .defer(d3.csv, "data/second-quarter");

Step 2) Wait for the files to load:

q.await(function(error, q1data, q2data) {

Step 3) Add the data to crossfilter:

    var ndx = crossfilter();
    ndx.add(q1data.map(function(d) {
        return { callTypes: d['Call Types'], 
                 callDesc: d['Call Description'],
                 callVol: d['Call Volume'],
                 quarter: 'Q1'};
    }));
    ndx.add(q2data.map(function(d) {
        return { callTypes: d['Call Types'], 
                 callDesc: d['Call Description'],
                 callVol: d['Call Volume'],
                 quarter: 'Q2'};
    }));

Step 4) Use your cross filter as you wish:

var timeDimension = ndx.dimension(function(d){
   return d.quarter;
});

dataTable
  ... //data table attributes

dc.renderAll();

Here is an example using this approach with the dc.js library: https://github.com/dc-js/dc.js/blob/master/web/examples/composite.html

Posterior

I just tried this the other week, and it worked like a charm. Thank you very much. – Tenebrous

Hello Martin, thanks for your answer. Please add all the needed js dependencies in your first block of code so newbies can get started swiftly. – Altaf