Query from minimongo of large number of records stucks and hangs browser
Asked Answered
B

5

10

I am building a page for admin in angular-meteor.

I have published all the records from a collection: "posts" and have taken the subscription of all the records on front end.

$meteor.subscribe('posts');

In the controller, if I select the cursors of all records from minimongo it works fine like:

$scope.posts = $meteor.collection(Posts);

But I want to display pagination, so for that I want limited records at a time like:

$scope.posts = $meteor.collection(function(){

        return Posts.find(

            {}, 
            {
                sort:   {'cDate.timestamp': -1},
                limit:  10
            }
        );
    });

It stucks with the query in minimongo. And the browser hangs.

"posts" collection contains only 500 records. It was working fine when I had 200 records.

Can anyone give me an idea whats wrong with my code and concepts?

EDIT:

Okay! It worked fine when I commented the $sort line from query like this:

$scope.posts = $meteor.collection(function(){

            return Posts.find(

                {}, 
                {
                    //sort:   {'cDate.timestamp': -1},
                    limit:  10
                }
            );
        });

But I need to sort the records. So what should I do now?

EDIT:

Also tried adding index to the sort attribute like this:

db.Posts.ensureIndex({"cDate.timestamp": 1})

Still same issue.

Beeswing answered 4/2, 2016 at 6:55 Comment(14)
Did you try adding an index on the cDate.timestamp field. Add index as mentioned in this post, #9730636 db.Posts.ensureIndex({"cDate.timestamp": 1})Mcburney
Did try adding an index. Doesn't work. Same issue. Browser is stuckBeeswing
I still think it is an index issue, since it is not having issue if you comment the sort line. You can try this suggestion #28732139 to make sure the query is using index. If it still doesn't solve the problem then sorry I also don't have idea.Mcburney
How about publishing only 10 (but sorted) of them?Waddle
@RadosławM So how will I show total records counter if I will have only 10 in minimongo.Beeswing
Create a method for that. Something like Meteor.methods({ postsCount: function () { return Posts.find({}, { fields: { _id: 1 } }).count(); } }).Waddle
The problem reminds me of this issue: github.com/meteor/meteor/issues/5633 I'm not familiar with the Angular integration in Meteor, but it could help to wait until the subscription is ready before calling find.Robalo
@chrisklaussner I don't see a possible solution to my problem in that ticket, yet still I tried to add indexes but no luck :(Beeswing
Sorting performances are horrible client-side as there is no indexing going on whatsoever (except for ID indexing). As such, any sorting should only be done on very small sets of data (such as 5 or 10 posts). When you run the query Meteor first tries to sort everything and only then select the 10 items you are looking for.Frumpy
@HassanSardar There is a hint in that ticket: "Publish functions send documents one at a time so that queries over the relevant collections need to be re-computed n times when you publish n documents." If you wait until all documents arrive at the client and call find afterwards, you won't trigger a repeated and unnecessary sorting of the collection. At least that worked on the reproduction repo provided in the issue. But I agree that you should also consider publishing fewer documents.Robalo
@chrisklaussner Well demanding fewer documents can help me out here, but the thing is what will happen when someday I will have to demand large number of documents for sorting and stuff. Issue exists still.Beeswing
It's strange you stuck at 500 records. Me, published > 1500 records and still fine. What kind of record you publish?Sheriff
@yudap You should try to sort your records on client. I published 3000 records with out sorting them on client, works like a charm, but as soon as you sort those record on client from minimongo, than you will see the magic.Beeswing
No. 1500 with sort on clientSheriff
M
3

Change your publication to accept a parameter called pageNumber like this

Meteor.publish('posts', function (pageNumber) {
   var numberOfRecordsPerPage = 10;
   var skipRecords = numberOfRecordsPerPage * (pageNumber - 1);
   return Post.find({
        "user_id": user_id
   }, {
        sort: { 'cDate.timestamp': -1 }
        skip: skipRecords, 
        limit: numberOfRecordsPerPage
   });
});

On client side, I didn't work with angular-meteor much. You can create a pageNumber property under your current scope using this.pageNumber or $scope.pageNumber. Update this pageNumber variable whenever your pagination page is clicked. Whenever this variable is changed, subscribe using the current page number.

If it is using standard blaze template, I would do it using a reactive var or session var in an autorun like this. In template html:

<template name="postsTemplate">
     <ul>
         <!-- you would want to do this list based on total number of records -->
         <li class="pagination" data-value="1">1</li>
         <li class="pagination" data-value="2">2</li>
         <li class="pagination" data-value="3">3</li>
     </ul>
</template>

In template js:

Template.postsTemplate.created = function () {
    var template = this;
    Session.setDefault('paginationPage', 1);
    template.autorun(function () {
       var pageNumber = Session.get('paginationPage');
       Meteor.subscribe('posts', pageNumber);
    });
}

Template.postsTemplate.events(function () {
    'click .pagination': function (ev, template) {
         var target = $(ev.target);
         var pageNumber = target.attr('data-value');
         Session.set('paginationPage', pageNumber);
     }
});

This way, you will have a maximum of 10 records at any point in time on the client, so it will not crash the browser. You might also want to limit the fields that you send to client using something like this

Meteor.publish('posts', function (pageNumber) {
   var numberOfRecordsPerPage = 10;
   var skipRecords = numberOfRecordsPerPage * (pageNumber - 1);
   return Post.find({
        "user_id": user_id
   }, {
        sort: { 'cDate.timestamp': -1 }
        skip: skipRecords, 
        limit: numberOfRecordsPerPage,
        fields: {'message': 1, 'createdBy': 1, 'createdDate': 1 } //The properties inside each document of the posts collection.
   });
});

And finally you will need the total number of records in posts collection on client side, to show the pagination links. You can do it using a different publication and using the observeChanges concept as mentioned in the official documentation here

// server: publish the current size of a collection
Meteor.publish("posts-count", function () {
  var self = this;
  var count = 0;
  var initializing = true;

  // observeChanges only returns after the initial `added` callbacks
  // have run. Until then, we don't want to send a lot of
  // `self.changed()` messages - hence tracking the
  // `initializing` state.
  var handle = Posts.find({}).observeChanges({
    added: function (id) {
      count++;
      if (!initializing)
        self.changed("postsCount", 1, {count: count});
    },
    removed: function (id) {
      count--;
      self.changed("postsCount", 1, {count: count});
    }
    // don't care about changed
  });

  // Instead, we'll send one `self.added()` message right after
  // observeChanges has returned, and mark the subscription as
  // ready.
  initializing = false;
  self.added("postsCount", 1, {count: count});
  self.ready();

  // Stop observing the cursor when client unsubs.
  // Stopping a subscription automatically takes
  // care of sending the client any removed messages.
  self.onStop(function () {
    handle.stop();
  });
});

// client: declare collection to hold count object
PostsCount = new Mongo.Collection("postsCount");

// to get the total number of records and total number of pages
var doc = PostsCount.findOne(); //since we only publish one record with "d == 1", we don't need use query selectors
var count = 0, totalPages = 0;

if (doc) {
    count = doc.count;
    totalPages = Math.ceil(count / 10); //since page number cannot be floating point numbers..
}

Hope this helps.

Mcburney answered 19/2, 2016 at 8:21 Comment(0)
B
2

Browser crashing because there is only so much data that it can load in it's cache before itself cashing out. To your question about what happens when you need to demand a large number of documents, take that process away from the client and do as much on the server through optimized publish and subscribe methods / calls. No real reason to load up a ton of documents in the browser cache, as minimongo's convenience is for small data sets and things that don't need to immediately sync (or ever sync) with the server.

Bight answered 16/2, 2016 at 6:12 Comment(3)
Ok than tell me how will I show the pagination on client? How will I show the reactive counter of all the records? Or how will I apply the filters which will search from complete database ?Beeswing
It's very possible. First, determine how you're going to subscribe to the posts in the first case. Sounds like you are moving away from minimongo (which is definitely in line with Meteor best practices).Bight
So if you're going to use a server/client Collection: Are you subscribing at the template level or the router level? What router will you be using? Are you publishing all your documents and then filtering with the subscription? Before you get into pagination these need to be handled first. Some people like to do the code for you here - I wanna see if you can provide us with some snippets of where you are at. If you need help with the above concepts, that's another story, but I think you can do it!Bight
U
2

you should sort on server side, you can find what you are looking for here

meteor publish with limit and sort

Ulysses answered 18/2, 2016 at 16:22 Comment(0)
A
2

You need to consider sort and limit strategies:

Sort on the server if you're extracting top values from a large set used by all clients. But usually better first filter by User who needs the data, and sort on the filtered collection. That will reduce the dataset.

Then Publish that sorted / limited subset to the client, and you can do more fine grained / sorting filtering there.

Asmodeus answered 19/2, 2016 at 1:51 Comment(0)
F
2

You should use server side limit instead of client side. That will make your app faster and optimize.

For more please check this link. link here

Fabian answered 23/2, 2016 at 14:2 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.