ExpressJS: Adding routes dynamically at runtime

Question

I want to be able to add new routes at runtime without restarting the server with NodeJS & ExpressJS. I made a similiar approach like in this article: https://alexanderzeitler.com/articles/expressjs-dynamic-runtime-routing/
Technically I'm able to add new files and logic at runtime likewise in the article, but the problem is that when no api route was matched I'll send a 404 JSON respond (as it is supposed to be).

I think the problem that I'm having is that my dynamically created routes are never reached, because static routes have priority over dynamically created routes. This means that the created routes will be mounted after error handling and therefore will never be reached. My Code in app.js

...

// Routes
app.use('/api/products', productRoutes);
app.use('/api/users', userRoutes);

...

/* This is where the dynamically created routes should be mounted */

// Error handling
app.use((req, res, next) => {
    const err = new Error('Not found');
    err.status = 404;
    next(err);
});

app.use((err, req, res, next) => {
    res.status(err.status || 500).json({error: {message: err.message}});
});

/* This is where the dynamic routes are mounted */

module.exports = app;

When I comment out the error handling I'm able to reach the routes which I created during runtime whereas with error handling I can only reach dynamically created routes after server restart which I want to avoid.
The problem is not solved with query params, because the dynamically added routes differ in logic, model properties, http methods/verbs and API endpoints. e.g.
GET/POST /api/{endpoint}
GET/POST /api/foo/{endpoint}
GET/PUT/DELETE /api/foo/bar/{endpoint}/:id

I think I basically need to either:
1) find a way to mount the dynamically created routes before the error handling - which I'm currently stuck at or
2) modify the route stack - which I have read is impractical, slow, bad practice and error prone
3) find an alternative solution

I hope someone can help me.
Thanks in advance

EDIT
Here is the code for the creation of new routes. The relevant endpoint is /api/databases/ in the POST method

const Database = require('../models/database');
const controller = require('./template/controller');
const creation = require('../Creation');

...

exports.createOne = (req, res, next) => {
  if (!creation.findFileInDirectory(`./backend/api/models/${req.body.name.singular}.js`) ||
      !creation.findFileInDirectory(`./backend/api/controllers/${req.body.name.singular}.js`) ||
      !creation.findFileInDirectory(`./backend/api/routes/${req.body.name.singular}.js`)) {
    controller.createOne(req, res, next, Database, {
      modelName: 'database',
    }, () => {
      //creation.createEndpoint(req.body.name, req.body.data, req.body.auth);
      creation.createEndpoint(req.body.name, req.body, req.body.auth);
    });
  } else {
    res.status(422).json({message: 'Endpoint exists already'});
  }
}

...

The controller in the snippet is just a modular controller file, which handles all of my CRUD Operations of all the endpoints of different models. Each route is split into models, controllers and routes to seperate and better maintain their logic.

In the POST method I first check whether the endpoint to be created already exists. If it does I respond with a 422 respond that the endpoint already exists. If it does not exist I create an entry mith my modular controller in the databases endpoint and create a model, controller & route for the endpoint which should be created.

The creation logic is the following:

const createEndpoint = (name, data, auth) => {
    createFile(`./backend/api/models/${name.singular}.js`, model.createModel(capitalize(name.singular), data), () => {
      createFile(`./backend/api/controllers/${name.singular}.js`, controller.createController({singular: capitalize(name.singular), plural: name.plural}, data.data), () => {
        createFile(`./backend/api/routes/${name.singular}.js`, route.createRoute({singular: capitalize(name.singular), plural: name.plural}, auth), () => {
          const app = require('../../app');
          mountEndpoints(name.singular, app);
        });
      });
    });
};

Here I basically pass along the data from the POST method to the model, controller & route file which are created asynchronously. When all files are created I mount the endpoint route to the app. The logic to mount the route is:

const mountEndpoints = (path, app) => {
  const module = require(`../routes/${path}`);
  app.use(`/api/${module.plural ? `${module.plural}` : `${path}s`}`, module);
}

A created route might look like the following:

const express   = require('express');
const router    = express.Router();
const checkAuth = require('../middleware/check-auth');

const ProductController = require('../controllers/product');

router.route('/')
    .get(ProductController.getAll)
    .post(checkAuth, ProductController.createOne);

router.route('/:id')
    .get(ProductController.getOne)
    .patch(checkAuth, ProductController.patchOne)
    .delete(checkAuth, ProductController.deleteOne);

module.exports = router;
module.exports.plural = 'products';

checkAuth includes some logic for authorization/authentication.

The code does pretty much what I want it to do except that I don't know how to handle the positioning of the route before the error handling.

Lithea

You probably need to show a bit more of the code where you are adding the dynamic routes... but I would add a router at runtime for the /path/ to handle the dynamic routes. Then add the dynamic routes to the router rather than app and they will be injected before the global app error handler – Abattoir

Try add most specific route before general route – Rafaelita

@Abattoir I edited my post and added some code and commentary. Can you elaborate a bit how to add the dynamic routes to the router instead the app? – Lithea

@Rafaelita In the end I pretty much only have one specific route which is the route to create other routes with the endpoint api/databases/. The general routes are not known before the server is started and should be added during runtime. – Lithea

@Lithea does the process stay the same? If yes you could api/:databaseName/something after the exact route. The problem with dynamic route on request is cleaning up . You can of coursd hack thourgh express but updates may screw up. – Rafaelita

@Rafaelita the creation process of every endpoint is the same, but the route to the endpoint, the logic of the controller, the CRUD methods & their logic, the model properties & their pre and post hooks are different. this depends on which data was sent in the post method. I don't really see the problem of cleaning up since the dynamic routes get files for each endpoint and on server restart I have logic implemented to search my route folder and mount all the routes within the folder so that they are all reachable again. Am I overlooking something regarding cleanup? – Lithea

The clean up i mean delete the unused routr,which is currently unsupported by express. Ignoring performance problem. It will also result in ambiguous behavior for request on the stack. – Rafaelita

@Rafaelita When creating routes during runtime there might obviously be the case that some of the routes won't be used anymore, but since I was having problems regarding mounting the routes during runtime I wasn't thinking too much about the cleanup regarding unsued/deleted routes. I found a response from an expressJS member regarding removing routes at runtime link He proposes to swap a router at runtime. Maybe it might help with creation of dynamic routes as well, but I'm not sure – Lithea

Answer 1

Express routes will be handled in creation order.

To add routes in specific locations after the app definition you can create a placeholder router and attach routes to that instead of modifying the app itself.

Express doesn't support deleting routes once they are defined, but you can replace an entire router.

Create an express router instance (or even another app if needed) to mount the dynamic endpoints on. Redefine the router whenever you want to change the routes (apart from additions to the end of the routers stack, which is supported by express).

// Routes
app.use('/api/products', productRoutes);
app.use('/api/users', userRoutes);

let dynamicApiRouter = null

export function setupDynamicRouter(route_configs) {
  dynamicApiRouter = new express.Router()
  // Add routes to dynamicApiRouter from `route_configs`
  for (const config of route_configs) {
    dynamicApiRouter[config.method](config.path, config.handler)
  }
}

app.use('/api', (req, res, next) => dynamicApiRouter(req, res, next))

// Error handling
app.use((req, res, next) => {
    const err = new Error('Not found');
    err.status = 404;
    next(err);
});

app.use((err, req, res, next) => {
    res.status(err.status || 500).json({error: {message: err.message}});
});

setupDynamicRouter() can be called at any time with one or a list of routes and handlers to setup:

const route_config = [
  {
    method: 'get',
    path: '/sales',
    handler: (req, res, next) => res.json({ ok: true }),
  },
  {
    method: 'post',
    path: '/sales',
    handler: (req, res, next) => res.json({ post: true }),
  },
])
setupDynamicRouter(route_config)

For the questions example "routes" setup, the /api path prefix now lives on the router mount in the parent app so can be removed from each router.use

const mountEndpoints = (path, router) => {
  const module = require(`../routes/${path}`);
  router.use(`/${module.plural ? `${module.plural}` : `${path}s`}`, module);
}

Abattoir

I had some time yesterday to tinker around and I managed to make it work with your suggestion. In my app.js file I used your approach and I exported the dynamicApiRouter to the file where my createEndpoint function is and mounted the module to the router. Maybe you can give me a tip how to handle deleteing unsued routes at runtime as well. @Rafaelita mentioned in a comment above that deleting/cleanup unsed routes may result in ambiguous behavior for requests on the stack. Other than that thank you for your help and I will definitely give koa a try. – Lithea

As AnonyMouze suggested, it's not a great idea to do that with express routes. You could write some middleware that handles the routing of your dynamic requests, then you could lookup a dynamic table rather than defining express routes – Abattoir

I have the logic for creating and mounting dynamic routes and their logic in an external file which kind of serves as my middleware. How would I define the logic for a dynamic table ? Wouldn't I use express for that as well in some form? – Lithea

My purpose for dynamic routes is that I want to create an api where different users can create an account and create different schemas for their own purpose. For example /api/user1/ wants to create a product schema with name, price and an item number and a schema for manufacturing them. So he would have /api/user1/products and /api/user1/manufacturer. User2 might have the same requirements for selling products, but his products needs different schema types. By creating dynamic routes I enable users to have a great flexibility for structuring their database management and personalisation. – Lithea

So instead of an app.get('/path') you app.use(dyanmicHandler) look at the path and headers and decide what function should handle the request. You might have to reimplement or reuse some of the helpers express provides but you will have more control – Abattoir

I ended up mounting all my routes to the dynamicApiRouter. I know that the app._router.stack is not supposed to be tinkered with, but what about the stack of the dynamicApiRouter? The dynamicApiRouter's stack is just the stack of configured routes. So when I mount my routes I can get the index and when I dynamically delete the route endpoint I can delete the corresponding files and the index in the stack like: dynamicRouter.stack.splice(deletedRouteIndex, 1); Is this a possible solution or is this still a hacky way and has possible drawbacks? – Lithea

Cool... I haven't messed with express internals much, but I'm going to guess the router logic is largely shared with the app so has the same caveats. It's probably safer to replace the router each time – Abattoir

I have seen that post before, but wasn't sure what to make of it, because I don't know what router(req, res, next) in the app.use function does.I ended up with this code: let router; const setupRouter = () => { router = new express.Router(); creation.mountEndpoints(./backend/api/routes/, router, true); } setupRouter(); app.use('/api', (req, res, next) => { router(req, res, next); }); I call setupRouter() whenever I add or delete a route at runtime. So I always create a new router when something changes. Not sure about adding a route like that regarding performance – Lithea

Adding and deleting routes at runtime works like that and there is no bad practice and/or hacky way invovled. (at least I think it isn't) – Lithea

I added that to the answer. app.use(router) and app.use((r,s,n) => router(r,s,n)) are functionally the same as they both use the listed function with the same args. The second form just gives you access to change the router variable function instead of express using the fixed function. – Abattoir

Thank you for explaining and editing your answer to address dynamic removal of routes as well. This was very useful. – Lithea

Answer 2

Just adding to the above answer, which is correct, you basically need either an express app or a Router instance to attach your routes to.

In my case I had a legacy Node.js app that was initialized asynchronously, meaning that the underlying express middleware was returned after the database connection was initialized internally.

Here myApp is the legacy app that have an init method with a callback that returns the express middleware instance after some time:

var express = require('express')
var myApp = require('my-app')

module.exports = (config) => {
  // create an app handle
  var handle = express()

  myApp.init(config, (err, instance) => {
    if (err) {}
    // mount lazily
    handle.use(instance)
  })

  // return immediately
  return handle
}

I called that wrapper module lazy-load.js and then I use it like this:

var express = require('express')
var lazyLoad = require('lazy-load')

express()
  .use('/a', lazyLoad(require('./config-a.json')))
  .use('/b', lazyLoad(require('./config-b.json')))
  .listen(3000)

I wanted to serve multiple instances of the same app but with different configuration for the different users.

As for the original question, again, as long as you keep reference to your handle middleware instance, initially attached to your server, you can keep adding new routes to it at runtime.

Dave