mongoexport aggregate export to a csv file

Question

I want to save the result from an aggregation into a csv file.

Using the mongo cmd line tool I can do this to get the results I want:

db.compras.aggregate({ $group : { _id : "$data.proponente", total : { $sum : "$price" } }}

How would I translate this into a mongoexport command that saves results into a csv?

Susy

FYI, mongoexport has --query option, but it doesn't support complex queries like yours. – Boyd

so there is no way to do it? – Susy

could you solve this problem? – Animalize

Answer 1

You can't run aggregate() queries through mongoexport. The mongoexport tool is intended for more basic data export with a query filter rather than full aggregation and data processing. You could easily write a short script using your favourite language driver for MongoDB, though.

Khamsin

Is this comment still true until this day ? because being new to mongodb, I'm currently trying to do exactly that – Anapest

This answer is still true in terms of mongoexport limitations, however there have been several major changes since May 2013: MongoDB 2.6 added support for aggregating into an output collection ($out) per the highly upvoted answer from Matt; the legacy mongo shell was deprecated for MongoDB 5.0 & replaced by mongosh which includes a full Node.js REPL; there's now a free MongoDB Compass GUI which also can also Export Aggregation Pipeline Results. – Khamsin

Answer 2

Slightly simpler option as of 2.6+ is to now add an $out step to your aggregate to put the results into a collection:

db.collection.aggregate( [ { aggregation steps... }, { $out : "results" } ] )

Then just use mongoexport as:

mongoexport -d database -c results -f field1,field2,etc --csv > results.csv

After that you might want to delete the temporary collection from the database so that it does not keep using unnecessary resources, and also to avoid confusion later, when you have forgotten why this collection exists in your database.

db.results.drop()

Schwejda

I tried this from within Robo 3T and it failed. However, when using the mongo shell directly this was like wonderful magic. – Algicide

It's worth all the efforts without the use of a client. But then I had faced a weird issue making use of this query - Mongo aggregate query results in less document with sorting which I doubt wouldn't have been the case with a client. (Though I am not completely sure of the comparison there.) – Alexio

flag --csv deprecated – Colvert

@Omar Aziz use: --type=csv – Impertinence

Answer 3

You can export to a CSV file with the following 3 steps:

1. Assign your aggregation results to a variable (reference):

   var result = db.compras.aggregate({ $group : { _id : "$data.proponente", total : { $sum : "$price" } }}
   

2. Insert a value of the variable to a new collection:

   db.bar.insert(result.toArray());
   

3. In terminal (or command line) export this bar collection to a CSV file:

   mongoexport -d yourdbname -c bar -f _id,total --csv > results.csv
   

...and you're done :)

Rale

Steps 1 and 2 can be combined as of MongoDB 2.6 using the $out operator on aggregate. See #13612528. – Annalist

This is really helpful for people still stuck with 2.4 on their servers. Upvoted. – Lordinwaiting

I believe step two should be db.bar.insert(result.toArray()); – Kob

Answer 4

If you don't want to store the results in a collection, you could also write directly to a CSV file from JavaScript using the print function. Save the following script to a file like exportCompras.js.

let cursor = db.compras.aggregate({ $group : 
  { _id : "$data.proponente", 
    total : { $sum : "$price" }
  }
});

if (cursor && cursor.hasNext()) {

  //header
  print('proponente,total');

  cursor.forEach(item => {
    print('"' + item._id + '",' + item.total);
    // printjson(item); -- if you need JSON
  });
}

From the command line, call >mongo server/collection exportCompras.js > comprasResults.csv --quiet

Updated from @M.Justin's comment to replace the previous while loop.

Bobbette

This could be simplified using .forEach rather than manually iterating over the cursor: cursor.forEach(item => print('"' + item._id + '",' + item.total)); – Anastomosis

For the current code, just need a ) after the forEach block above – Beecher

Answer 5

Take the following and save it on the mongo server somewhere:

// Export to CSV function
DBCommandCursor.prototype.toCsv = function(deliminator, textQualifier) 
{
    var count = -1;
    var headers = [];
    var data = {};

    deliminator = deliminator == null ? ',' : deliminator;
    textQualifier = textQualifier == null ? '\"' : textQualifier;

    var cursor = this;

    while (cursor.hasNext()) {

        var array = new Array(cursor.next());

        count++;

        for (var index in array[0]) {
            if (headers.indexOf(index) == -1) {
                headers.push(index);
            }
        }

        for (var i = 0; i < array.length; i++) {
            for (var index in array[i]) {
                data[count + '_' + index] = array[i][index];
            }
        }
    }

    var line = '';

    for (var index in headers) {
        line += textQualifier + headers[index] + textQualifier + deliminator;
    }

    line = line.slice(0, -1);
    print(line);

    for (var i = 0; i < count + 1; i++) {

        var line = '';
        var cell = '';
        for (var j = 0; j < headers.length; j++) {
            cell = data[i + '_' + headers[j]];
            if (cell == undefined) cell = '';
            line += textQualifier + cell + textQualifier + deliminator;
        }

        line = line.slice(0, -1);
        print(line);
    }
}

Then you can run the following commands via the shell or a GUI like Robomongo:

load('C:\\path\\to\\your\\saved\\js\\file')
db.compras.aggregate({ $group : { _id : "$data.proponente", total : { $sum : "$price" } }}.toCsv();

Pediment

Answer 6

By the same logic, in my case, I want the output as JSON data into out.json

1- Create file.js on local, not on the server

date = new Date("2019-09-01");
query = {x: true, 'created': {$gte: date}};
now = new Date();
userQuery = {'user.email': {$nin: [/\.dev$|@test\.com$|@testing\.com$/]}};
data = db.companies.aggregate([
    {$match: query},
    {$sort: {x: 1, created: 1}},
    {$lookup: {as: 'user', from: 'users', localField: 'creator', foreignField: '_id'}},
    {$addFields: {user: {$arrayElemAt: ['$user', 0]}}},
    {$match: userQuery},
    {$project: {"_id": 1, "name": 1, "user.email": 1, "xyz": 1}}
]).toArray();

print(JSON.stringify(data));

2- Run

mongo server/collection file.js > out.json --quiet

3- Delete header and check out.json

[
  {
    "_id": "124564789",
    "xyz": [
      {
        "name": "x",
        "value": "1"
      },
      {
        "name": "y",
        "value": "2"
      }
    ],
    "name": "LOL",
    "user": {
      "email": "[email protected]"
    }
  },
....
....
}]

Roller

Answer 7

Try this for query-

mongoexport -d DataBase_Name -c Collection_Name --type=csv --fields name,phone,email -q '{_bank:true,"bank.ifsc":{$regex:/YESB/i}}' --out report.csv

Scrunch

Answer 8

I tried to run the export as suggested, but I'm getting csv flag deprecated

so here is a working snippet

mongoexport -d database_name -c collection_name -f field1,field_2 --type=csv > results.csv

Optionally, you can add a filter like the following

mongoexport -h 0.0.0.0 -d database_name -c collection_name --query '{"$or": [{"condition_one": true},{"condition_two": true}]}' --type=csv --fields "field1,field_2" > out.csv

Colvert