Shared models between two Rails apps - what is the ideal solution for Workflow?

Question

I am currently working on a Rails 3 project that is divided up into four parts:

• The public facing website
• The administration website/backend
• The models
• The API for third party data access

As the models are shared between the three key components I want to keep them away from being in one main project, however each part needs access to the models, but I don't want to repeat the code and have different versions everywhere.

Currently I have the model code in a gem, and in each project's Gemfile I am referencing them with the following line:

gem "my_models", :path => "../my_models/"

However when I deploy to our test servers for my co-workers to evaluate the system on I need to pull the models from an external repository, so I swap out the above line with the following:

gem "my_models", :git => "[email protected]:username/my_models.git"

This in its self works well, but its quite clunky in terms of 'versions' (i.e. I need to bump the version every time I wish to deploy the changes to the test servers), switch the line over to use git instead of local, and make sure that I'm pushing the files properly.

Previously I was using a shared git submodule, but this was just as awkward.

I would rather not build everything into one mega-project, as these tend to become monstrous and difficult to maintain, and I would also like to separate concerns if possible, so any changes I make to the administration site doesn't have much of a chance to impact the other components - obviously the models have the potential to cause issues, but that is a risk I have considered and understand.

What would people out there suggest when it comes to something like this? Or, am I going about it completely the wrong way?

Some additional background:

This app is a rewrite of an existing website which followed the model of 'lump everything into the one project' - unfortunately there are two issues here:

1. The app was badly developed - I inherited this project and when I first picked it up the load times were ~2 minutes per page with a single user - this has since been reduced but still has issues throughout
2. We are currently at our capacity limit of the current site and we anticipate that we will need to take on more load in the next 6 months - however scaling out with an 'all in one' app means we'll be wasting resources on scaling out the back end of the site which doesn't need it.

Essentially there are two things I want to separate - the Front end (being the public website and the API) and the back end - everything I know about software development tells me that combining all this together is not an ideal solution (and past history shows me that splitting these two is a good move in terms of ensuring front end performance).

Perhaps I need to look at this from another angle - keep the models in each project, and instead of sharing them between projects have a cut-down subset of functionality for each functional area (i.e. the backend needs to know who created a post, but the front end doesn't really care about that, so omit that logic when reading in the model).

Vladimir

What didn't you like about the git submodule flow? That is the solution that comes to mind instantly. It solves version control and centralizes the code. – Satang

Creating a "mega-project" probably isn't a good idea. SOA and microservices might be a better approach. – Foushee

Answer 1

drop the models project(put models into one of other parts, i'd suggest whatever you consider "more important"), put all projects into single repository(separate project folders) and make symlinks to models/libs/apis/whatever

your code is highly coupled together and you often need to make changes to few projects at once(like updating models and updating APIs that use them, etc)

one nice thing about single-repo-symlink setup is that your commits will be less fragmented and will usually represent full feature implementation - easier to track bugs, read history and maintain codebase

also when you deploy you don't need to read from many repositories - one less point of failure right there

release process is also simpler with such model as branch will now hold the scope of all projects

there are some drawbacks like symlinks dont work that well on windows and whatnot but for me it works perfectly

Hippogriff

obviously i don't mean that you should put all your projects into single repo. but i would put there everything that shares your models(because it probably shares much more than that). – Hippogriff

This actually sounds like a really nice solution. I'll try this out and then give feedback/upvote/bounty :) – Devaluate

thanks, don't hesitate to ask any questions - i'll try to help as much as i can – Hippogriff

I'm working on a project with a similar setup, so I found this answer helpful. However, I ran into a brick wall when trying to deploy the "public facing website" on Heroku. The public site app symlinks models and the database.yml file from the admin app. Heroku uses git for deployment, and git doesn't follow symlinks, so I'm bit stuck. I could use hard links, but this makes pushing and pulling awkard (duplicating data). If anyone has suggestions, I'm all ears. – Azarcon

@Azarcon git wors with symlinks just fine(i'm not sure about older versions though). assuming you have projects admin and web try this: cd web/app; ln -s ../../admin/app/models ./models - this works for me – Hippogriff

@Hippogriff yes, you are correct. Git tracks the symlink files, but when I deploy, the symlinks, which reference an undeployed project, are uploaded and not the models themselves. Example: consider the two projects admin and web stored under the project directory (also single git repo). admin has the models directory. web links like you mentioned. I only want to deploy web, not admin. I deploy the web subtree with git subtree push --prefix web heroku master, but in this case, I am deploying a symlink (models) that doesn't go anywhere (because admin isn't deployed). – Azarcon

@Azarcon i'm not familiar with heroku deploy process, my capistrano scripts are setup to fetch whole repo and rsync related folder which follows symlinks correctly, try to customize heroku's process to do the same if it's possible. good luck :) – Hippogriff

Ahh, makes sense. Heroku is deployment is strictly reliant on git (it's not like you have a server you can ssh into), so I don't think it's possible. That doesn't stop me from deploying somewhere else though :) Thanks for the reply. – Azarcon

Curious if @Hippogriff or anyone has found a way to make this symlink approach work with heroku. In this boat now, and it's looking like a private models gem is the only way to go. – Valencia

Answer 2

You can create a mountable engine that contains the shared models and create a gem out of it. This will handle the name spacing issues elegantly. Other nice aspect here is you get to share your assets also.

Watch this railscast for more details.

Imparipinnate

Answer 3

You'll still have to manage the 'versions' by pushing changes that need to be tested to a remote repo, but you can use the new local config of Bundler 1.2

http://gembundler.com/man/bundle-config.1.html#LOCAL-GIT-REPOS

This way it will pick up your local commits and you won't have to keep change your Gemfile upon deployment.

Laryngotomy

Answer 4

I know that this is not an solution for your particular problem. But I really suggest you to merge all projects into one. It is very usual to have all this parts in one application and there is no overhead. I think there is no not-awkward solution for this problem.

Overriding

Completely agree. This is usually something that's done when an application becomes huge and you want to break it into separate services. – Clinical

In the previous iteration of this project everything was all in one, with administration being in a different namespace. The issue with this is that it runs under the same process, so if there is something going on in the admin or API it affects the rest of the site. – Vladimir

I don't agree. Merging everything into one will lead to a monolithic application which requires updating whenever any single part of the application changes, regardless of it's relevance. For instance, you would have to bring down your API to deploy a CSS change on the website. – Delay

@Neil wouldnt that depend on how you deploy? These are after all just files on a hard drive one can be replaced without necessarily having to replace them all, Capistrano default does not do this, but in theory you could only replace changed files. – Magistracy

@MatthewSavage You should be running multiple instances of the site through Passenger or something similar. One user should not be able to DOS the site through a long-running request. – Guardafui

In a lot of cases you will want to have a separate API that does not have the entire Rails middleware stack with it (to decrease the workload). You'll have a hard time putting that together with a full web stack. – Devaluate

I've found it quite useful to partition an app as he describes. And have done something very similar. This makes it much easier to scale appropriately as each section of the app is being used. – Laryngotomy

Answer 5

Take look at Git subtree.

This may work for you..

http://igor-alexandrov.github.io/blog/2013/03/28/using-git-subtree-to-share-code-between-rails-applications/

OR

You can write Rake task..

Example:-

namespace :sync do

  desc 'Copy common models and tests from Master'
  task :copy do
   source_path = '/home/project/src-path'
   dest_path = '/home/project/dest-path'

   # Copy all models & tests
   %x{cp #{source_path}/app/models/*.rb #{dest_path}/app/models/}
   %x{cp #{source_path}/spec/models/*_spec.rb #{dest_path}/spec/models/}

   # Database YML
   %x{cp #{source_path}/config/database.yml #{dest_path}/config/database.yml}

end

See the below link.

http://hiltmon.com/blog/2013/10/14/rails-tricks-sharing-the-model/

Hijack

Answer 6

Does your project have enough code coverage? If it does, I would try to separate the logic where it makes sense, and if a model is used in different projects, just pick one that fits best and write an API on top of that.

Then you could use that API to access those models (preferably using something like ActiveModel) on the other project. You would still have a simple CRUD, but all the core model logic would be handled externally.

Be sure to think well before splitting them up, though. You want to keep your domain tight on each app you create out of the Behemoth you want to torn apart.

Regarding engines:

I have used Engines for the same issue and it does help, but I also had to change my Gemfile to either point to a local path when developing, pushing the gem, then pulling it on the current project, which is the behavior you're not fond of.

Defenestration