How can I "cache" a mongoDB/Mongoose result to be used in my Express.js views and routes

Question

What I'm trying to achieve is some sort of way to cache results of a mongoDB/Mongoose query that I can use in my views and routes. I'd need to be able to update this cache whenever a new document is added to the collection. I'm not sure if this is possible and if it is then how to do it, due to how the functions are asynchronous

This is currently what I have for storing the galleries, however this is executed with every request.

app.use(function(req, res, next) {
  Gallery.find(function(err, galleries) {
    if (err) throw err;  
      res.locals.navGalleries = galleries;
      next();
  });
});

This is used to get gallery names, which are then displayed in the navigation bar from a dynamically generated gallery. The gallery model is setup with just a name of the gallery and a slug

and this is part of my EJS view inside of my navigation which stores the values in a dropdown menu.

<% navGalleries.forEach(function(gallery) { %>
  <li>
    <a href='/media/<%= gallery.slug %>'><%= gallery.name %></a>
  </li>
<% }) %>

The website I'm working on is expected to get hundreds of thousands of concurrent users, so I don't want to have to query the database for every single request if not needed, and just update it whenever a new gallery is created.

Rociorock

Answer 1

Take a look at cachegoose. It will allow you to cache any query you want and invalidate that cache entry each time a new gallery is created.

You will need something like this:

const mongoose = require('mongoose');
const cachegoose = require('cachegoose');

cachegoose(mongoose); // You can specify some options here to use Redis instead of in-memory cache

app.get(function(req, res, next) {
    ...

    Gallery
        .find()
        .cache(0, 'GALLERY-CACHE-KEY')
        .exec(function(err, galleries) {
            if (err) throw err;  

            res.locals.navGalleries = galleries;

            next();
    });

    ...
});

app.post(function(req, res, next) {
    ...

    new Gallery(req.body).save(function (err) {
        if (err) throw err;

        // Invalidate the cache as new data has been added:
        cachegoose.clearCache('GALLERY-CACHE-KEY');
    });

    ...
});

Although you could do something simpler caching the results manually in a variable and invalidating that cache when new galleries are added, I would advise you to take a look at that package instead.

Narvik

Perfect! I think this is just what I need. Very easy setup as well. Thank you very much – Rociorock

Answer 2

I've created modern library for handling cache clearing automatically. In short it's more advanced and faster than cachegoose, because it's also caching Mongoose Documents instances in memory.

Speedgoose has autoclearing plugin, which you can set on a given schema, So it's not only ttl-based. It works on mongoose document events to clear results related to the document. So if the result of query XYZ is cached, and it was containing the record that was edited/removed, it will be removed from the cache. Same with adding and removing documents from schema. Those events might affect every cached result in the collection. So they are clearing the cache in the scope of a given model. I should be rather enough in 90% of cases.

In near future it will have many more features. And it's pretty easy to setup!

https://www.npmjs.com/package/speedgoose

1. Setup

import {applySpeedGooseCacheLayer} from "speedgoose";
import mongoose from "mongoose";

applySpeedGooseCacheLayer(mongoose, {
  redisUri: process.env.REDIS_URI
})

1. Cache something!

//Caching both - query result, and instances of related Mongoose Documents in memory. It supports sort, select,project...etc.
model.find({}).sort({fieldA : 1}).cacheQuery()

//Caching only result
model.find({}).lean().cacheQuery()

//Works also with aggregation
model.aggregate([]).cachePipeline()

Sharkskin

Good idea, but using Redis there adds an extra unneeded layer, IMHO, because Redis works as a cache only in memory. It crashes when super heavily used and consumes CPU and memory (referring to the Redis app itself here). I don't see the difference then. You can have your own memory cache to manage the size easily. – Weber

In that case, Redis works as a shared memory layer. So it's rather needed, to share results between nodejs cluster instances. Objects instances are cached straight in given cluster instance in Map(). In near future, I will add feature to use more 'cache clients'. Redis was just for the initial release. So for 'super heavy' cases that your Redis couldn't handle properly, you would be able to provide your own client. The main point is that I would like to keep share results between instances. But I'm open for suggestions – Sharkskin

As I know memcached perform better than redis, but then I have to keep both. Memcached as storage, and Redis for pub/sub. But I would give a choice to the user. – Sharkskin

> The main point is that I would like to keep share results between instances. And that's a good idea, imho. Also, a pub/sub would be good to have as well. One more challenge I see is to revalidate cache data as specifying only TTL won't be sufficient in some cases. – Weber

Speedgoose has autoclearing plugin, which you can set on a given schema, So it's not only ttl-based. It works on mongoose document events to clear results related to the document. So if the result of query XYZ is cached, and it was containing the record that was edited/removed, it will be removed from the cache. Same with adding and removing documents from schema. Those events might affect every cached result in the collection. So they are clearing the cache in the scope of a given model. I should be rather enough in 90% of cases. – Sharkskin

That sounds brilliant. One last thing. Maybe I've overlooked it, but I'd love to see their support to-file-cache so instead in-memory all data will be read/written on the disk. I think SSD today's day is pretty good for caching. – Weber

It's on project roadmap :) maybe i will add it in this month. Thank you for your suggestions @CezaryTomczyk – Sharkskin

Answer 3

In case you using typescript try ts-cache-mongoose

// On your application startup
import mongoose from 'mongoose'
import cache from 'ts-cache-mongoose'

// In-memory example 
const cache = cache.init(mongoose, {
  engine: 'memory',
})

// Redis example
const cache = cache.init(mongoose, {
  engine: 'redis',
  engineOptions: {
    host: 'localhost',
    port: 6379,
  },
})

mongoose.connect('mongodb://localhost:27017/my-database')

// Somewhere in your code
const users = await User.find({ role: 'user' }).cache('10 seconds').exec()
// Cache hit
const users = await User.find({ role: 'user' }).cache('10 seconds').exec()

const book = await Book.findById(id).cache('1 hour').exec()
const bookCount = await Book.countDocuments().cache('1 minute').exec()
const authors = await Book.distinct('author').cache('30 seconds').exec()

const books = await Book.aggregate([
  {
    $match: {
      genre: 'fantasy',
    },
  },
  {
    $group: {
      _id: '$author',
      count: { $sum: 1 },
    },
  },
  {
    $project: {
      _id: 0,
      author: '$_id',
      count: 1,
    },
  }
]).cache('1 minute').exec()

Mere

For some reason I get this error: Error when evaluating SSR module mongoose.ts: failed to import "ts-cache-mongoose" |- ReferenceError: exports is not defined in ES module scope Any idea what I could be doing wrong? – Accelerant