MongoDB queries are taking 2-3 seconds from Node.js app on Heroku

Question

I am having major performance problems with MongoDB. Simple find() queries are sometimes taking 2,000-3,000 ms to complete in a database with less than 100 documents.

I am seeing this both with a MongoDB Atlas M10 instance and with a cluster that I setup on Digital Ocean on VMs with 4GB of RAM. When I restart my Node.js app on Heroku, the queries perform well (less than 100 ms) for 10-15 minutes, but then they slow down.

Am I connecting to MongoDB incorrectly or querying incorrectly from Node.js? Please see my application code below. Or is this a lack of hardware resources in a shared VM environment?

Any help will be greatly appreciated. I've done all the troubleshooting I know how with Explain query and the Mongo shell.

var Koa = require('koa'); //v2.4.1
var Router = require('koa-router'); //v7.3.0

var MongoClient = require('mongodb').MongoClient; //v3.1.3

var app = new Koa();
var router = new Router();

app.use(router.routes());

//Connect to MongoDB
async function connect() {
    try {
        var client = await MongoClient.connect(process.env.MONGODB_URI, {
            readConcern: { level: 'local' } 
        });
        var db = client.db(process.env.MONGODB_DATABASE);

        return db;
    }
    catch (error) {
        console.log(error);
    }
}

//Add MongoDB to Koa's ctx object
connect().then(db => {
    app.context.db = db;
});

//Get company's collection in MongoDB
router.get('/documents/:collection', async (ctx) => {
    try {
        var query = { company_id: ctx.state.session.company_id }; 

        var res = await ctx.db.collection(ctx.params.collection).find(query).toArray();

        ctx.body = { ok: true, docs: res };
    }
    catch (error) {
        ctx.status = 500;
        ctx.body = { ok: false };
    }
});

app.listen(process.env.PORT || 3000);

UPDATE

I am using MongoDB Change Streams and standard Server Sent Events to provide real-time updates to the application UI. I turned these off and now MongoDB appears to be performing well again.

Are MongoDB Change Streams known to impact read/write performance?

Cicatrix

Are you able to check network latency or at least to make sure Heroku and DO instances are in the same region? – Carbarn

Do you have an index on company_id? – Imputation

@Carbarn I have not checked latency between Heroku and DO. How can I do that? – Cicatrix

@AdamHarrison I've tried with and without an index on company_id. No effect because the database has so few docs I think. – Cicatrix

What's spec of your heroku instance? – Overanxious

Try first to get to where the problem really is: 1) Connect directly to the MongoDB instance using e.g. mongo.exe or Robo3T or similar and see how your query behaves. Is it slow? Then there's some MongoDB related problem which I personally doubt or the network causes problems between client and server. 2) Turn on MongoDB query logging. 3) Measure the network throughput and latency without any MongoDB queries (skip the query in your code and return something directly). 4) Try updating your MongoDB driver to the latest version. 5) Add tracing output and measure time between statements. – Eyeleteer

@MạnhQuyếtNguyễn It's running on Hobby level dynos on Heroku. I do not think that is the problem though because there is almost zero load on the application and memory and CPU are both fine. – Cicatrix

You might want to include the change stream code that you think is causing problem. It may be that your usage is not correct or may be there some bug in your mongo version. – Schlessel

Answer 1

Change Streams indeed affect the performance of your server. As noted in this SO question.

As mentioned in the accepted answer there,

The default connection pool size in the Node.js client for MongoDB is 5. Since each change stream cursor opens a new connection, the connection pool needs to be at least as large as the number of cursors.

const mongoConnection = await MongoClient.connect(URL, {poolSize: 100});

(Thanks to MongoDB Inc. for investigating this issue.)

You need to increase your pool size to get back your normal performance.

Chinchilla

Thanks for the info. For now, I've just disabled Change Streams and my performance is back to where it was. – Cicatrix

@CoreyQuillen how did you disable change stream. can you explain please – Leitmotiv

Answer 2

I'd suggest you do more log works. Slow queries after restarted for a while might be worse than you might think.

For a modern database/web app running on a normal machine, it's not very easy to encounter with performance issues if you are doing right. There might be a memory leak or other unreleased resources, or network congestion.

IMHO, you might want to determine whether it's a network problem first, and by enabling slow query log on MongoDB and logging in your code where the query begins and ends, you could achieve this.

If the network is totally fine and you see no MongoDB slow queries, that means something goes wrong in your own application. Detailed logging might really help where query goes slow.

Hope this would help.

Interpol