Rails 4 has_and_belongs_to_many doesn't work properly with includes statement

Question

Recently I ran into mystical bug with rails 4 and HABTM relation. first of all my Gemfile:

source 'https://rubygems.org'
gem 'rails', '~> 4.1.6'
gem 'pg'

Next. my Models:

class User < ActiveRecord::Base
end

class Teacher < User
  has_and_belongs_to_many :resources, foreign_key: :user_id
end

class Resource < ActiveRecord::Base
  has_and_belongs_to_many :teachers, association_foreign_key: :user_id
end

Raw DB data:

select * from resources;
 id |         created_at         |         updated_at
----+----------------------------+----------------------------
  1 | 2014-10-13 08:24:07.308361 | 2014-10-13 08:24:07.308361
  2 | 2014-10-13 08:24:07.889907 | 2014-10-13 08:24:08.156898
  3 | 2014-10-13 08:24:08.68579  | 2014-10-13 08:24:08.884731
  4 | 2014-10-13 08:24:09.997244 | 2014-10-13 08:24:10.205753
(4 rows)

select * from users;
 id |         created_at         |         updated_at         |  type
----+----------------------------+----------------------------+---------
 13 | 2014-10-13 08:24:01.086192 | 2014-10-13 08:24:01.086192 | Teacher
 12 | 2014-10-13 08:24:00.984957 | 2014-10-13 08:24:00.984957 | Teacher
  2 | 2014-10-13 08:23:59.950349 | 2014-10-16 08:46:02.531245 | Teacher
(3 rows)

select * from resources_users;
 user_id | resource_id
---------+-------------
      13 |           1
       2 |           2
      12 |           3
       2 |           4
(4 rows)

Finally the bug:

➜  rails_test  bundle exec rails c
Loading development environment (Rails 4.1.6)
2.1.2 :001 > Resource.all.includes(:teachers).map(&:teachers).map(&:to_a)
  Resource Load (0.6ms)  SELECT "resources".* FROM "resources"
  SQL (1.3ms)  SELECT "resources_users".*, "resources_users"."user_id" AS t0_r0, "resources_users"."resource_id" AS t0_r1, "users"."id" AS t1_r0, "users"."created_at" AS t1_r1, "users"."updated_at" AS t1_r2, "users"."type" AS t1_r3 FROM "resources_users" LEFT OUTER JOIN "users" ON "users"."id" = "resources_users"."user_id" AND "users"."type" IN ('Teacher') WHERE "users"."type" IN ('Teacher') AND "resources_users"."resource_id" IN (1, 2, 3, 4)
 => [
[#<Teacher id: 13, created_at: "2014-10-13 08:24:01", updated_at: "2014-10-13 08:24:01", type: "Teacher">],
[],
[],
[]]

As you see only first array of teachers returned in collection. However SQL generated by Rails is correct and returns all data:

SELECT "resources_users".*, "resources_users"."user_id" AS t0_r0, "resources_users"."resource_id" AS t0_r1, "users"."id" AS t1_r0, "users"."created_at" AS t1_r1, "users"."updated_at" AS t1_r2, "users"."type" AS t1_r3 FROM "resources_users" LEFT OUTER JOIN "users" ON "users"."id" = "resources_users"."user_id" AND "users"."type" IN ('Teacher') WHERE "users"."type" IN ('Teacher') AND "resources_users"."resource_id" IN (1, 2, 3, 4);
 user_id | resource_id | t0_r0 | t0_r1 | t1_r0 |           t1_r1            |           t1_r2            |  t1_r3
---------+-------------+-------+-------+-------+----------------------------+----------------------------+---------
      13 |           1 |    13 |     1 |    13 | 2014-10-13 08:24:01.086192 | 2014-10-13 08:24:01.086192 | Teacher
       2 |           2 |     2 |     2 |     2 | 2014-10-13 08:23:59.950349 | 2014-10-16 08:46:02.531245 | Teacher
      12 |           3 |    12 |     3 |    12 | 2014-10-13 08:24:00.984957 | 2014-10-13 08:24:00.984957 | Teacher
       2 |           4 |     2 |     4 |     2 | 2014-10-13 08:23:59.950349 | 2014-10-16 08:46:02.531245 | Teacher
(4 rows)

Did anyone faced such problem before? I can't understand what's going on here.

P.S. If you do Resource.all.includes(:teachers).map { |r| r.reload.teachers } the result is correct. However it removes sense from include at all and provides N+1 problem.

UPDATE: One more finding worth mentioning. If I remove STI everything works fine.

Gasteropod

Just wanna mention here that has_and_belongs_to_many is sort of the old school way for many-to-many relationships. A better alternative is has_many :through. Source – Platitudinous

Just wanna mention that there's nothing "old school" about has_and_belongs_to_many, you should use the type of association that is most fitting for your task. For a lot of things, HABTM is perfectly fine. – Autarchy

Answer 1

I recreated those ActiveRecord models and database records in Rails 4.1.6 with pg gem and saw the correct behavior:

irb(main):017:0> Resource.all.includes(:teachers).map(&:teachers).map(&:to_a) Resource Load (0.6ms) SELECT "resources".* FROM "resources" SQL (6.9ms) SELECT "resources_users".*, "resources_users"."id" AS t0_r0, "resources_users"."resource_id" AS t0_r1, "resources_users"."user_id" AS t0_r2, "users"."id" AS t1_r0, "users"."type" AS t1_r1, "users"."created_at" AS t1_r2, "users"."updated_at" AS t1_r3 FROM "resources_users" LEFT OUTER JOIN "users" ON "users"."id" = "resources_users"."user_id" AND "users"."type" IN ('Teacher') WHERE "users"."type" IN ('Teacher') AND "resources_users"."resource_id" IN (1, 2, 3, 4) => [[#<Teacher id: 13, type: "Teacher", created_at: "2015-11-05 07:02:59", updated_at: "2015-11-05 07:02:59">], [#<Teacher id: 2, type: "Teacher", created_at: "2015-11-05 07:02:20", updated_at: "2015-11-05 07:02:32">], [#<Teacher id: 12, type: "Teacher", created_at: "2015-11-05 07:03:50", updated_at: "2015-11-05 07:03:50">], [#<Teacher id: 2, type: "Teacher", created_at: "2015-11-05 07:02:20", updated_at: "2015-11-05 07:02:32">]]

Evyn

Answer 2

Same bug here with Rails 4.1.6 and pg, but I can get the correct behavior by not removing the resources_users id field in the migration:

def change
  #create_table :resources_users, id: false do |t|
  create_table :resources_users do |t|
    t.integer :resource_id
    t.integer :user_id
  end
end

Also eager_load works in both cases (with or without id on the join table) and you have the benefits of a single SQL query:

Resource.eager_load(:teachers).map(&:teachers).map(&:to_a)

output:

irb(main):002:0> Resource.eager_load(:teachers).map(&:teachers).map(&:to_a)
SQL (1.0ms)  SELECT "resources"."id" AS t0_r0, "resources"."created_at" 
AS t0_r1, "resources"."updated_at" AS t0_r2, "users"."id" AS t1_r0, 
"users"."type" AS t1_r1, "users"."created_at" AS t1_r2, 
"users"."updated_at" AS t1_r3 FROM "resources" LEFT OUTER JOIN 
"resources_users" ON "resources_users"."resource_id" = "resources"."id" 
LEFT OUTER JOIN "users" ON "users"."id" = "resources_users"."user_id" 
AND "users"."type" IN ('Teacher')
=> [[#<Teacher id: 1, type: "Teacher", created_at: "2015-11-05 
15:02:33", updated_at: "2015-11-05 15:02:33">], [#<Teacher id: 2, type: 
"Teacher", created_at: "2015-11-05 15:02:33", updated_at: "2015-11-05 
15:02:33">], [#<Teacher id: 2, type: "Teacher", created_at: "2015-11-05 
15:02:33", updated_at: "2015-11-05 15:02:33">], [#<Teacher id: 3, type: 
"Teacher", created_at: "2015-11-05 15:02:33", updated_at: "2015-11-05 
15:02:33">]]

Microcyte