How to implement server side rendering datatable, Using node and mongo db?

Question

So i have one user collection(mongo DB) which consists millions of user.

I m using nodejs as backend, angular js as frontend and datatable for displaying those users.

But datatable Load all users in one api call which load more then 1 million user.

This makes my API response two slow.

I want only first 50 users then next 50 then so on....

Server stack = node js + angular js + mongo DB

Thanks

Aileen

Take a look at the answer to this post https://mcmap.net/q/237182/-mongodb-paging – Reede

I want pagination in side data table. – Aileen

Answer 1

If you are using datatable with huge amount of data you should consider using server side processing functionnality.

Server side processing for datatable is described here : https://datatables.net/manual/server-side

But if you feel lazy to implement this on your server you could use third parties like :

• https://github.com/vinicius0026/datatables-query
• https://github.com/eherve/mongoose-datatable

Hope this helps.

Edy

Glad to hear that :-) – Edy

Answer 2

The way to solve you client trying to fetch users from your server(and DB) and then rendering them to a datatable is done using pagination. There a few ways of solving pagination which i have seen, let's assume you are using REST.

One way of doing this is having your API ending with:

/api/users?skip=100&limit=50

Meaning, the client will ask your server for users(using default sorting) and skipping the first 100 results it finds and retrieving the next 50 users.

Another way is to have your API like this(I don't really like this approach):

/api/users?page=5&pageSize=50

Meaning, the client will pass which page and how many results per page it wants to fetch. This will result in a server side calculation becuase you would need to fetch users from 250-300.

You can read on pagination a lot more on the web.

Having said that, your next issue is to fetch the desired users from the database. MongoDB has two functions for using skip and limit, which is why I like the first API better. You can do the query as follows:

users.find().skip(50).limit(50)

You can read more about the limit function here and the skip function here

Intoxicative

But How can I implement this in datatable >> next button? – Aileen

Lets say the user views the first page which shows users from 0 to 50, and then clicks on the next button, you would then do another call to the server requesting users from 50 to 100. You db query would look like users.find().skip(50).limit(50) – Intoxicative

@MaheshGareja Did this help you? – Intoxicative

Nope! still confuse on server side logic. – Aileen

Please tell me what confuses you and I'll help you. Basically your API call should tell you which users to fetch. – Intoxicative

Answer 3

First Thing you need in to add skip and limit to you mongo query like this Model.find().skip(offset).limit(limit)

then the next thing you have to do is enable server side processing in datatables

If you are using javascript data-table then this fiddle will work for you http://jsfiddle.net/bababalcksheep/ntcwust8/

For angular-datatables

http://l-lin.github.io/angular-datatables/archives/#/serverSideProcessing

One other way if you want to send own parameters

  $scope.dtOptions = DTOptionsBuilder.newOptions()
        .withOption('serverSide', true)
        .withOption('processing', true)

        .withOption('ajax', function (data, callback, settings) {
            // make an ajax request using data.start and data.length

            $http.post(url, {
                draw: draw,
                limit: data.length,
                offset: data.start,
                contains: data.search.value
            }).success(function (res) {
                // map your server's response to the DataTables format and pass it to
                // DataTables' callback
                draw = res.draw;

                callback({
                    recordsTotal: res.meta,
                    recordsFiltered: res.meta,
                    draw: res.draw,
                    data: res.data
                });
            });
        })

you will get the length per page and offset as start variable in data object in the .withOption('ajax' , fun...) section and from there you can pass this in get request as params e.g. /route?offset=data.start&limit?data.length or using the post request in above example

On hitting next button in table this function will automatically trigger with limit and start and many other datatable related value

Leprose

@Mahesh Gareja please update if you found any answer usefull – Leprose

what is the error? are you using angular datatables? if you can share some angularjs code then i can help – Leprose

At frontend side there is no issue. Problem is backend because I m using native mongo query and also I want datatable searching enable – Aileen

then share your backend code. Datatable searching is also enabled through server side processing . in data object you will recieve a string taht you type in datatable to search and from here you can pass that string to your post request – Leprose

Answer 4

@mahesh when loading page create 2 variables lets say skipVar=0 and limit when user clicks on next send *skipVar value key skip

var skipVar =0
on page load skip=skipVar&limit=limit
on next button 
skipVar=skipVar*limit
and send Query String as 
skip=skipVar&limit=limit

Glossematics

Answer 5

I found mongo-datatable module very easy to use.

To retrieve the data from MongoDB it requires DataTables' sent-paramters should be in the form of JSON object that is the query of express request.

Here is pseudo code. (check example by author here)

const MongoDataTable = require('mongo-datatable');
const mongodb = require('mongodb');
const MongoClient = mongodb.MongoClient;

MongoClient.connect('mongodb://localhost/database', function(err, client) {
  const db = client.db('database');
  new MongoDataTable(db).get('collection', options, function(err, result) {
    if (err) {
      // handle the error
    }   
    // result is required output
    // send to client 
    res.json(result);
    // or return a promise in your function
    resolve(result);
  }); 
});

This module has some extra options (showAlertOnError, customQuery and caseInsensitiveSearch) which are required for any practical application development.

Apathy