Include all existing fields and add new fields to document

Question

I would like to define a $project aggregation stage where I can instruct it to add a new field and include all existing fields, without having to list all the existing fields.

My document looks like this, with many fields:

{
    obj: {
        obj_field1: "hi",
        obj_field2: "hi2"
    },
    field1: "a",
    field2: "b",
    ...
    field26: "z"
}

I want to make an aggregation operation like this:

[
    {
        $project: {
            custom_field: "$obj.obj_field1",
            //the next part is that I don't want to do
            field1: 1,
            field2: 1,
            ...
            field26: 1
        }
    },
    ... //group, match, and whatever...
]

Is there something like an "include all fields" keyword that I can use in this case, or some other way to avoid having to list every field separately?

Chancey

This feature is coming in the next major release 2.6. You can try it on the unstable dev branch - use "$$ROOT" to refer to the entire incoming document. See details in this ticket: jira.mongodb.org/browse/SERVER-5916 – Eunaeunice

This issue is also raised at #20497999, with some other useful and different answers. – Proton

Answer 1

In 4.2+, you can use the $set aggregation pipeline operator which is nothing other than an alias to $addFieldsadded in 3.4

The $addFields stage is equivalent to a $project stage that explicitly specifies all existing fields in the input documents and adds the new fields.

db.collection.aggregate([
    { "$addFields": { "custom_field": "$obj.obj_field1" } }
])

Ballman

Any idea how to use it in C# driver? seems it does not exist – Zettazeugma

I'm not familiar with the C# driver but $addFields is new in MongoDB 3.4 which is supported by the C# driver version 2.5+ – Ballman

I have been searching for a way to do this in C# driver, and so far it looks like the way to do it would be using something like IAggregateFluent<TResult>.AppendStage(new JsonPipelineStageDefinition<TInput, TOutput>("{ $addFields : { myField: 'myValue' }}") – Sheer

I discovered that it can even replace a field if you specify the same field name – Aesculapius

Answer 2

You can use $$ROOT to references the root document. Keep all fields of this document in a field and try to get it after that (depending on your client system: Java, C++, ...)

 [
    {
        $project: {
            custom_field: "$obj.obj_field1",
            document: "$$ROOT"

        }
    },
    ... //group, match, and whatever...
]

Rutty

But this will create an embedded doc called document an option to created one merged doc would have been nicer... – Steviestevy

Does anyone know of the solution to pass through all the key value pairs without creating the embedded document? – Banns

Hold your breath. MongoDB 3.4 will come with $addFields aggregation stage that does just this. See https://mcmap.net/q/137905/-mongodb-project-retain-previous-pipeline-fields-duplicate. – Week

This seems to be the best current solution, as you can (at least in JS) then use the spread operator to rebuild your original object with the new custom_field attached. const newObj = { ...result.document, custom_field: result.custom_field } – Mons

Answer 3

To add new fields to your document you can use $addFields

from docs

and to all the fields in your document, you can use $$ROOT

db.collection.aggregate([

{ "$addFields": { "custom_field": "$obj.obj_field1" } },
{ "$group": {
        _id : "$field1",
        data: { $push : "$$ROOT" }
    }}
])

Orianna

He wants all fields in root. This adds all fields under documents array. – Delisadelisle

Answer 4

>>> There's something like "include all fields" keyword that I can use in this case or some another solution?

Unfortunaly, there is no operator to "include all fields" in aggregation operation. The only reason, why, because aggregation is mostly created to group/calculate data from collection fields (sum, avg, etc.) and return all the collection's fields is not direct purpose.

Delk

...because aggregation is mostly created to group/calculate data from collection fields... Best answer in fact! – Rasbora

you have a collection posts with _id, title, body, likes fields. The likes field is an array of user _id who like the post. How could you list all posts with all the _id, title, body, likeCount? Returning all fields is a direct purpose in this case. – Yeh

Answer 5

As of version 2.6.4, Mongo DB does not have such a feature for the $project aggregation pipeline. From the docs for $project:

Passes along the documents with only the specified fields to the next stage in the pipeline. The specified fields can be existing fields from the input documents or newly computed fields.

and

The _id field is, by default, included in the output documents. To include the other fields from the input documents in the output documents, you must explicitly specify the inclusion in $project.

Fervidor

Answer 6

according to @Deka reply, for c# mongodb driver 2.5 you can get the grouped document with all keys like below;

var group = new BsonDocument
{
 { "_id", "$groupField" },
 { "_document", new BsonDocument { { "$first", "$$ROOT" } } }
};

ProjectionDefinition<BsonDocument> projection = new BsonDocument{{ "document", "$_document"}};
var result = await col.Aggregate().Group(group).Project(projection).ToListAsync();

// For demo first record 
var fistItemAsT = BsonSerializer.Deserialize<T>(result.ToArray()[0]["document"].AsBsonDocument);

Heywood