N+1 problem in mongoid

Question

I'm using Mongoid to work with MongoDB in Rails.

What I'm looking for is something like active record include. Currently I failed to find such method in mongoid orm.

Anybody know how to solve this problem in mongoid or perhaps in mongomapper, which is known as another good alternative.

Arcadian

Please consider re-assessing your answers now that some time has passed. I believe there are some mis-conceptions around this topic. – Gradation

Answer 1

Update: it's been two years since I posted this answer and things have changed. See tybro0103's answer for details.

---

Old Answer

Based on the documentation of both drivers, neither of them supports what you're looking for. Probably because it wouldn't solve anything.

The :include functionality of ActiveRecord solves the N+1 problem for SQL databases. By telling ActiveRecord which related tables to include, it can build a single SQL query, by using JOIN statements. This will result in a single database call, regardless of the amount of tables you want to query.

MongoDB only allows you to query a single collection at a time. It doesn't support anything like a JOIN. So even if you could tell Mongoid which other collections it has to include, it would still have to perform a separate query for each additional collection.

Tusche

This is not correct. Rails' :include options does not do a join. It eager loads. Eager loading is necessary for both SQL and Mongo. It prevents you from querying the same collection over and over while you're in a loop, and instead, it queries the collection once before you start the loop. – Gradation

Consider a view with a list of posts and under each post is a list of comments. If you don't eager load you'll have to query the comments collections once per post (N+1). Fortunately, Mongoid does have includes()... see my answer. – Gradation

Answer 2

Now that some time has passed, Mongoid has indeed added support for this. See the "Eager Loading" section here:
http://docs.mongodb.org/ecosystem/tutorial/ruby-mongoid-tutorial/#eager-loading

Band.includes(:albums).each do |band|
  p band.albums.first.name # Does not hit the database again.
end

I'd like to point out:

1. Rails' :include does not do a join
2. SQL and Mongo both need eager loading.
3. The N+1 problem happens in this type of scenario (query generated inside of loop):

.

<% @posts.each do |post| %>
  <% post.comments.each do |comment| %>
    <%= comment.title %>
  <% end %>
<% end %>

Looks like the link that @amrnt posted was merged into Mongoid.

Gradation

just to add a comment. You should enabled identify map to make it works mongoid.org/en/mongoid/docs/identity_map.html – Calumnious

Mongoid includes supports eager loading direct relations, but doesn't support eager loading two-level relations or eager loading a polymorphic belongs_to. If you need this features, I created a library that extends Mongoid to support those use cases. – Sweetandsour

@Calumnious mongoid5 - no more identity map. so not needed anymore. – Pewit

@HertzelGuinness yup, after 3 years, lot of things change.. right! – Calumnious

Answer 3

Although the other answers are correct, in current versions of Mongoid the includes method is the best way to achieve the desired results. In previous versions where includes was not available I have found a way to get rid of the n+1 issue and thought it was worth mentioning.

In my case it was an n+2 issue.

class Judge
  include Mongoid::Document

  belongs_to :user
  belongs_to :photo

  def as_json(options={})
    {
      id: _id,
      photo: photo,
      user: user
    }
  end
end

class User
  include Mongoid::Document

  has_one :judge
end

class Photo
  include Mongoid::Document

  has_one :judge
end

controller action:

def index
  @judges = Judge.where(:user_id.exists => true)
  respond_with @judges
end

This as_json response results in an n+2 query issue from the Judge record. in my case giving the dev server a response time of:

Completed 200 OK in 816ms (Views: 785.2ms)

The key to solving this issue is to load the Users and the Photos in a single query instead of 1 by 1 per Judge.

You can do this utilizing Mongoids IdentityMap Mongoid 2 and Mongoid 3 support this feature.

First turn on the identity map in the mongoid.yml configuration file:

development:
  host: localhost
  database: awesome_app
  identity_map_enabled: true

Now change the controller action to manually load the users and photos. Note: The Mongoid::Relation record will lazily evaluate the query so you must call to_a to actually query the records and have them stored in the IdentityMap.

def index
  @judges ||= Awards::Api::Judge.where(:user_id.exists => true)
  @users = User.where(:_id.in => @judges.map(&:user_id)).to_a
  @photos = Awards::Api::Judges::Photo.where(:_id.in => @judges.map(&:photo_id)).to_a
  respond_with @judges
end

This results in only 3 queries total. 1 for the Judges, 1 for the Users and 1 for the Photos.

Completed 200 OK in 559ms (Views: 87.7ms)

How does this work? What's an IdentityMap?

An IdentityMap helps to keep track of what objects or records have already been loaded. So if you fetch the first User record the IdentityMap will store it. Then if you attempt to fetch the same User again Mongoid queries the IdentityMap for the User before it queries the Database again. This will save 1 query on the database.

So by loading all of the Users and Photos we know we are going to want for the Judges json in manual queries we pre-load the data into the IdentityMap all at once. Then when the Judge requires it's User and Photo it checks the IdentityMap and does not need to query the database.

Grime

Why wouldn't you just do Judge.where(...).includes(:photo, :user)? – Gradation

This was a solution before 'includes' was added into mongoid and can still be beneficial for understanding. – Grime

"Although the other answers are correct there is no method of joining queries across collections I have found a way to get rid of the n+1 issue" ...seems to state that this answer solves the N+1 problem while others answers do not. However, using includes does solve the N+1 problem and it had been implemented in Mongoid more than a year before this was posted, as much older answers have referred to it. – Gradation

I agree. I didn't say I posted this solution before 'includes' was added. I stated this was a solution before 'includes' was added. and I posted it for educational purposes anyway. I also don't think anything it mentions states that the other answers do not solve the problem. Almost the opposite I open with "The other answers are correct." – Grime

Answer 4

ActiveRecord :include typically doesn't do a full join to populate Ruby objects. It does two calls. First to get the parent object (say a Post) then a second call to pull the related objects (comments that belong to the Post).

Mongoid works essentially the same way for referenced associations.

def Post
    references_many :comments
end

def Comment
    referenced_in :post
end

In the controller you get the post:

@post = Post.find(params[:id])

In your view you iterate over the comments:

<%- @post.comments.each do |comment| -%>
    VIEW CODE
<%- end -%>

Mongoid will find the post in the collection. When you hit the comments iterator it does a single query to get the comments. Mongoid wraps the query in a cursor so it is a true iterator and doesn't overload the memory.

Mongoid lazy loads all queries to allow this behavior by default. The :include tag is unnecessary.

Radiometeorograph

This is also incorrect. Consider the case where you're displaying a list of posts with a list of comments under each post. Using post.comments while you're looping through posts will generate a new query each time unless you use Rails' :include. Fortunately, Mongoid does have includes()... see my answer. – Gradation

Answer 5

This could help https://github.com/flyerhzm/mongoid-eager-loading

Gyro

Answer 6

You need update your schema to avoid this N+1 there are no solution in MongoDB to do some jointure.

Cipolin

Answer 7

Embed the detail records/documents in the master record/document.

Airman

Answer 8

In my case I didn't have the whole collection but an object of it that caused n+1 (bullet says that).

So rather than writing below which causes n+1

quote.providers.officialname

I wrote

Quote.includes(:provider).find(quote._id).provider.officialname

That didn't cause a problem but left me thinking if I repeated myself or checking n+1 is unnecessary for mongoid.

Propst