Mongoose Populate returning null or undefined

Question

I'm sorry if this is a n00b question, I've been searching Google & Stack for hours now and I've got to ask!

I have two schemas, User and Story, shown below. I am trying to reference the User for a Story using the Ref option to Populate in a Query - I've using mySQL before and so wanted to try to replicate a JOIN statement. Whenever I try to use populate I just get the objectID returned, or null (shown below).

Edited 12 Nov to fix hardcoded IDs & add console data

story-schema.js

var mongoose = require('mongoose'),
Schema = mongoose.Schema,
User = require('./user-schema');

var StorySchema = new Schema({  
title: { type: String, required: true },  
author_id: { type: Schema.Types.ObjectId, ref: 'User' },
summary: { type: String, required: true },
rating: { type: String, required: true }
});

module.exports = mongoose.model('Story', StorySchema, 'stories');

user-schema.js

var mongoose = require('mongoose'),
Schema = mongoose.Schema;

var UserSchema = new Schema({
    username: { type: String, required: true, index: { unique: true } },
    is_admin: {type: Boolean, required: true, default: false }
});

save - id hardcoded for example

app.post('/api/new_story', function(req, res){
var story;
story = new Story({
    title: req.body.title,
    author_id: mongoose.Types.ObjectId(req.params._id), 
            /* ex of req.params._id: 527fc8ff241cdb8c09000003*/
    summary: req.body.summary,
    rating: req.body.rating
});

story.save(function(err) {
    if (!err) {
        return console.log("created");
    } else {
        return console.log(err);
    }
});
return res.send(story);
});

example user when logged in terminal

{
"__v" : 0,
"_id" : ObjectId("527fc8ff241cdb8c09000003"),
"is_admin" : false,
"username" : "ted"
}

example story when logged in terminal

{
"title" : "Test Story",
"author_id" : "527fc8ff241cdb8c09000003",
"summary" : "Test summary",
"rating" : "12",
"_id" : ObjectId("52827692496c16070b000002"),
"__v" : 0
}

queries

//other mongoose/app includes above
User = require('./config/schema/user-model'),
Story = require('./config/schema/story-model');

// data.author_id = 527fc8ff241cdb8c09000003 
// data.author_id.username = undefined
app.get('/api/query/:id', function (req, res){
return Story.findOne({_id:req.params.id})
.populate( { path: 'User' } )
.exec(function (err, data) {
            console.log(data.author_id);
            console.log(data.author_id.username);
          if (err) {
            return res.json({error:err})
          }
        })
});

// data.author_id = null 
app.get('/api/query2/:id', function (req, res){
return Story.findOne({_id:req.params.id}) //_id hardcoded for example
.populate( 'author_id' )
.exec(function (err, data) {
            console.log(data.author_id);
          if (err) {
            return res.json({error:err})
          }
        })
});

In the first query I get the author_id I already saved back, which kind of makes sense as that's what I saved - but I access the username.

In the second query I can't even access the author_id I've already saved.

Edit: I can run a normal GET query fine without the 'populate'

What I'd like to happen

Is to be able to access the author information from the story - this is more like a proof of concept. Eventually I'd like to reference the Story _id in the the User model as there can be many Stories to a User, but only one User per Story but thought I'd start here first.

Philae

Answer 1

Since your example includes raw Object IDs not even quoted as valid javascript strings, it's hard to understand what is a mistake in your question vs a mistake in your code, but mostly your second query looks like it should work. I suspect you are mixing up your ObjectId values. You are using the same ObjectId for your story and your user and that won't work:

   author_id: mongoose.Types.ObjectId(528149b38da5b85003000002),

That's the same ID you use to lookup the story. So I think the fix is:

1. Make sure you have a valid user record in the users collection
2. Make sure your test story record's story.author_id matches that
3. Make sure you are looking up the story by the story Id and not the author_id.

If you still can't get this working, code the entire thing in a single .js file with both schemas, both models, and some code to create the records then do the queries, and post that.

Instrumentation

Hi I've updated the examples to not have the hardcoded data - it was a copy and paste error in my effort to make things clearer (fail). I can return a valid user record when trying to return just the user (1). The author_id matches the user's _id, however even though I'm trying to save it as a ObjectID it's showing as a string in the console. Could this be it? – Philae

OK, so yes I think the fact that story.author_id is a String and not an ObjectId is a problem. It's a data problem you need to just fix by loading and re-saving that record. My guess is you created that story either before your mongoose schema was finalized or via the mongo shell or whatever, but essentially the data is not compliant with the schema, which is always a risk with this type of database. – Instrumentation

Ok, I created a new record but still no luck, then I tried a fresh Schema, fresh collection just in case but no joy. After that I stuck everything in one file as in your first comment and it works (!) with the original /api/query2/:id etc - I think it must be something to do with the separate files being required in each other - I found this answer link but that didn't work for me either when splitting them back up. So right now, 1 file it is! – Philae

Ah, yes you probably created a circular require graph which is a no-no in node. Don't use require to reference each model in each other, just use the string model name in your schema refs. – Instrumentation

Answer 2

I was stuck with this problem for a week. After trying everything and making sure that I had no circular references, I decided to reset my DB.

After removing all collections, and storing them again, all my relations managed by mongoose's populate() worked.

It is so frustrating because this issue is not documented and I lost 20-30 hours trying to make a stupid collection reference work.

Evolutionary

We are facing a similar issue now. We didnt make any changes to the version of mongoose, still returning null for some of the populate() cases which were working fine for last 4 years – Cloudlet

same here i am using the Model.populate() method.... – Cissie

Answer 3

Is it possible that you forget to include a "ref" field in your Schema? I just encountered this issue and after adding a ref field to my Schema it works immediately without any need to reset the database. Hope this information helps.

I found an example documented here: https://mongoosejs.com/docs/3.0.x/docs/populate.html. Although it is a documentation for version 3.0.x, I found it helpful even for version 5.5.5 which is now I am using.

Crocus

After hours and hours of debugging and digging through the ridiculously awful Mongoose docs, this answer finally gave a hint about the need for refs. Thanks Leo. FML. – Knuth

Answer 4

I stuck in this problem for 5-6 hours and found a very simple solution for that. In this case you can see here :

    {
   "title" : "Test Story",
   "author_id" : "527fc8ff241cdb8c09000003",
   "summary" : "Test summary",
   "rating" : "12",
   "_id" : ObjectId("52827692496c16070b000002"),
   "__v" : 0
   }

in author_id value '527fc8ff241cdb8c09000003' is basically a "string" not ObjectId. This happens when you restore the database from json or csv and during this process ObjectId converts into the string. In My case I have dumped data using Python and try to retrieve the data using express. So to solve this problem simply I updated the data using express and now "Populate" is working try updating the data :

 const { ObjectId }  = require('bson');
    app.get('/update-data', (req,res)=>{
         Story.find()
        .exec((err,posts)=>{
            for(let value of posts)
            {
             User.findOne({ _id : value.author_id })
             exec((err,userData)=>{
                      Story.update({ _id : value._id }, {  author_id : ObjectId(userData._id) })
              .exec((error, resData)=>{
                   console.log(resData);
                 });
               })
            
            }
            res.send('Data Updated, try populating data again');
        });
    });
    

and that's it, after reestablishing the relation with user try populating the field again.

Update 1 :
I forgot to mention required package for 'ObjectId()' must include this

const { ObjectId }  = require('bson');

Update 2 : After digging more deeper I found that updating is not the solution. I found that you need to include the all model you want to populate. for example : there is a countryModel which is connected to user. then in your code country model and user model both should be included like this way:

var userModel = require('../<path to model>/userModel');
var countryModel = require('../<path to model>/countryModel');

however you are not using countryModel but this need to be initiate. And for country Model code should be : countryModel.js

const mongoose = require('mongoose');

const countrySchema = mongoose.Schema({
country_name : String,
flag_image : String,
country_code : String
});

//user model 
var countryModel = mongoose.model('countries', countrySchema);
module.exports = countryModel;

Gaga

Answer 5

I managed to solve this problem when declared the ref as model class reference instead of string with the name of the model.

// before
let Schema = mongoose.Schema({
    name: String,
    author: { type: Schema.Types.ObjectId, ref: 'Author' }
});

//after
let Schema = mongoose.Schema({
    name: String,
    author: { type: Schema.Types.ObjectId, ref: Author }
});

Bessel

You should include / require the class that you want to ref to, this is just an example. Don't just downvote the answer, and try to analyse the example. – Bessel

This is an important point: Mongoose docs are VERY unclear about whether/when to use strings vs. live object references. – Radborne

The live reference will always work. The string will work if the module setup and hierarchy is okay. – Bessel

Answer 6

If your data is very small then Try clearing your data from tables related to populate and main one

Grampus

Answer 7

Sometimes this error occurs when you try to populate something that doesn't exist anymore in the database. I encountered this error because of this reason.

Assiniboine

Answer 8

I stumbled upon this and the problem was referencing a deleted document. reset your DB if you can or double check the referenced id

Destroyer

Answer 9

I had a similar issue. Typo error on collection name. I had collection with name Currency, but correct collection name is Currencies

Dapplegray

Hi, this is a comment, not an answer... Be sure to read help on how to answer – Quahog