What's the right way to set up a development environment on OS X with Docker?

Question

Intro

I can't figure out a good way to set up a development environment on OS X using Docker and Boot2Docker. The problem I'm hitting is how to manage the source code so that:

1. I can modify the code on OS X using the tools (text editor, IDE, git, etc) I already have installed.
2. Those modifications are reflected in the Docker container so if I re-run tests or refresh a webpage, I can see my changes immediately.

In theory, this should be easy to do by mounting my source code as a volume:

docker run -it -v /path/to/my/source/code:/src some-docker-image

Unfortunately, this has two major issues that make it completely unusable on OS X:

Issue #1: Mounted volumes on VirtualBox (which use vboxsf) are extremely slow

For example, here is how long it takes Jekyll to compile my homepage if the source code is part of the Docker image:

> docker run -it brikis98/yevgeniy-brikman-homepage:v1 bash

root@7aaea30d98a1:/src# time bundle exec jekyll build

[...]

real    0m7.879s
user    0m7.360s
sys     0m0.600s

Here is the exact same Docker image, except this time, I mount the source code from OS X:

> docker run -it -v $(pwd):/src brikis98/yevgeniy-brikman-homepage:v1 bash

root@1521b0b4ce6a:/src# time bundle exec jekyll build

[...]

real    1m14.701s
user    0m9.450s
sys     0m3.410s

Issue #2: File watching is broken

The default watch mechanisms in SBT, Jekyll, and grunt use technologies such as inotify, which do not work if they are running in a Docker container and the changes are made in OS X to a mounted folder.

Workarounds I tried

I searched for solutions (including all the ones on SO) and tried out a few of them, but have not found a successful one:

1. I switched Boot2Docker to use NFS, but it was just as slow.
2. I tried Vagrant + NFS, and that was also just as slow.
3. I tried a Samba mount, but the folder always showed up empty in the Docker container.
4. I tried to use the Unison file system, which worked briefly to sync files, but then kept showing connection errors.
5. I enabled polling in Jekyll, but that significantly increased the delay until my changes were picked up.
6. I tried Dinghy, a "faster, friendlier Docker on OS X with Vagrant" and got some improvement. Instead of Jekyll compilation being 10-15x slower, it was 2-3x slower. That's better, but still not quite usable.

Has anyone found a solution that actually works and allows you to productively develop code with Docker and OS X?

Update: a solution at last!

I have finally found a solution that seems productive using Boot2Docker + rsync. I've captured the details on how to set this up in my own answer as well as an open-source project called docker-osx-dev.

Cypripedium

You've tried the official Docker installer for OS X right along with NFS? AFAIK this is not an issue limited to Docker on OS X but also Vagrant based development on OS X with larger codebase(s) (we have a similar issue but with Vagrant). I've found NFS to be the only viable and acceptable solution. – Colpitis

@JamesMills: I followed the official instructions to install Docker and Boot2Docker. Are there official instructions for setting up NFS? I only found them in a GitHub gist, and after using them, it didn't seem faster. How did you setup NFS? – Cypripedium

Have you seen github.com/boot2docker/boot2docker/issues/64 ? – Colpitis

The right way to work with Docker is to run Linux natively instead of OS X, or do all your development work inside a Linux VM. The "boot2docker" integration is a big ugly hack that does nothing but sow confusion and disappointment. – Camden

@JamesMills: after reading through all 70 comments, I found this script that is supposed to change VirtualBox to use NFS. After running it, performance for mounted folders is still just as bad. I updated the original question with example timings. – Cypripedium

@larsks: That is not helpful. – Cypripedium

@yevgeniy-brikman The truth hurts! And honestly, I do not find doing my development work in a VM particularly problematic. I find it much simpler than mucking about with OS X/virtualbox integration. – Camden

@yevgeniy-brikman I believe NFS does fix performance issues when trying to share file system resources from host to vm. I've been down this road with Vagrant-based development. I just don't have the opportunity to fully test this and answer the question for you at this time :) My tip would be to keep at it until you solve it :) "It should be possible!" (tm) :) – Colpitis

@JamesMills: Did you follow the same procedure as in this script to enable NFS? Or did you do something else? – Cypripedium

I use a boot2docker iso that included guest additions and nfs support. Trouble is I can't find an updated version of this ISO anymore. – Colpitis

FWIW; IHMO the recommended way to do any kind of development is to develop in isolation (barding any OS X + Docker performance problems with sharing host file system resources). i.e: I typically develop on my host anyway (which is Linux) and use Docker (in any environment) for testing, integration and deployment(s). – Colpitis

I have set up several docker dev environments in OS X, nfs is the way to go with vagrant, I ditched boot2docker a long time ago, and make sure your time is in sync. – Marrin

Sorry if this is out of left field, but if the goal is to develop on OSX leveraging an external code repo, is there possibly a more 'native' approach using XCode? Maybe some custom connector or repo-side Mercurial/SVN gateway? I just wanted to raise the idea since all comments/solutions seem to stick to OPs context of working almost exclusively with non-native components and one or more of these components might play nicer when hooked in with more reliable native/well-tested solutions for osx – Englut

Another possible workaround would be to stick with NFS and move the output of Jekyll outside of NFS. I ran few benchmarks on my machine as POC: gist.github.com/m1keil/cf2c489a51e5bdf6ee7c – Ridgeway

@m1keil: Interesting idea. So NFS performance is mostly limited by writes? That would make sense. The problem, however, is that the only reason to use NFS over rsync is so you have two-way sync, but if you're putting the output of code running on the VM in an non-NFS folder, then you lose that benefit. – Cypripedium

That's true, but sometimes you don't need high performance bi-directional sync (like in your case). I left more detailed comment in your blog post. – Ridgeway

I'm using coreos-vagrant with optimized nfs share options. My setup: jverdeyen.be/docker/how-php-symfony-coreos-docker . I'm also using dnsdock to resolve domains. For performance I also create data-container with a tmpfs (ram disk) mount shared accross containers. – Legendre

As long as github.com/docker/for-mac/issues/77 is an open issue there is not much hope for a workaround-less solution. – Fasto

Answer 1

I've decided to add my own answer with the best solution I've found so far. I'll update this if I find better options.

Best solution so far

The best solution I've found for setting up a productive development environment with Docker on OS X is: Boot2Docker + Rsync. With rsync, build times in a Docker container are on par with running the build directly on OSX! Moreover, the file watcher code does not need polling (inotify works since rsync uses normal folders), so hot reload is almost as fast.

There are two ways to set it up: an automated install and a manual install.

Automated install

I've packaged all the steps for setting up Boot2Docker with Rsync into an open source project called docker-osx-dev. The code is a bit rough, but I've been successfully using it for several weeks to easily switch between 3 projects with 3 different tech stacks. Try it out, report bugs, and submit some PRs! Also, see my blog post, A productive development environment with Docker on OS X for more info.

Manual setup

1. Install Boot2Docker: brew install boot2docker.
2. Run Boot2Docker, but with VirtualBox shared folders disabled: boot2docker init && boot2docker start --vbox-share=disable.
3. Run boot2docker shellinit and copy the environment variables it prints out into your ~/.bash_profile file.
4. Install rsync on the Boot2Docker VM: boot2docker ssh "tce-load -wi rsync".
5. Create the base folders you need on the Boot2Docker VM and set permissions correctly for them. For example, if you'll be syncing the /foo/bar folder from OS X, you need to create /foo/bar on the Boot2Docker VM: boot2docker ssh "mkdir -p /foo/bar && chown -R docker /foo/bar".
6. Run rsync to sync the files to the Boot2Docker VM: rsync --archive --rsh="ssh -i $HOME/.ssh/id_boot2docker -o StrictHostKeyChecking=no" /foo/bar docker@dockerhost:/foo. Check the rsync docs for various settings you may want to enable, such as using --exclude .git to exclude the .git folder when syncing.
7. Use a file watcher to keep files in sync. For example, you could use fswatch (brew install fswatch) piped into rsync.
8. At this point, you should be able to use docker run to fire up your Docker container and use the -v flag to mount the folder you're syncing: docker run -v /foo/bar:/src some-docker-image.
9. Update the code on OS X as usual. Changes should propagate very quickly using rsync, the normal file watcher code should pick up the changes as usual (ie, using inotify), and the build should run fast because all the files are "local" to the container.
10. If you need to test a running website, run the boot2docker ip command to find out what IP it's on.

Cypripedium

Thanks for sharing! When they say rsync is "one-way only" does it mean I can't use the OS X file system to share files between two containers? Example: container 1 watches source files & compiles a binary, container 2 is used to run the compiled binary (using Haskell in this example). – Verminous

@NicolasHery: My understanding is that rsync will copy changes from OS X to the Docker container, but not the other way around. Therefore, any files generated by the Docker container (e.g. a compiled binary) won't be visible in OS X. However, if those files are generated into a folder marked as a VOLUME, then you could give another container access to that volume using the --volumes-from flag. I haven't tried that yet, but I suspect it would work. – Cypripedium

Great answer. You could create a driver for docker-machine (github.com/docker/machine) that does most of the boilerplate for you. – Conservatory

@dom: I like that idea, but do you know how to create a driver for docker-machine? Is a pull request into the repo the only way or is it possible to create a driver externally? – Cypripedium

Is this tutorial still valid for a fresh 1.9.1 version on Windows? Can I use it or maybe Docker had already a new solution for this "problem"? – Centreboard

@JohnSam: I've only tested this solution on OS X. I'm not sure what it would take to get it working in windows, but some of the alternatives I listed have windows support, so you could try those. – Cypripedium

Is your solution still up to date? Seems like docker does not require to use boot2docker anymore – Lues

@sheepwalker: I believe Docker still uses boot2docker under the hood, but they package it up inside of Docker Toolbox. docker-osx-dev should probably be updated to use Docker Toolbox directly, but it works fine for now as-is. – Cypripedium

I saw that Boot2Docker is deprecated in favor of Dockertools. So my assumption is that I can still use the rsync trick in order to avoid mounting folders? Correct? – Gomulka

@tester: Docker Tools uses Boot2Docker under the hood, so yes, the same idea still works. – Cypripedium

I independently developed a similar solution also using rsync to push local file and directory changes to the data volume (either classic data volumes or data volume containers). This works well for Docker Toolbox scenarios on a Mac and should also work for Windows folks via Bash for Windows. See the examples at github.com/JeNeSuisPasDave/asd-sync-src-to-container and a discussion of the issue at develves.net/blogs/asd/2016-11-26-still-using-docker-toolbox. I've been using this mechanism on a variety of projects to great success. Most recently tested with Docker 1.12.3. – Rodman

Answer 2

Update: Now that docker for mac is in beta with non-hack functionality, going that route may be a lot more reasonable for local development without a essay's worth of hacks and workarounds.

Don't. I know that's not the answer you are probably hoping for, but take an honest evaluation of the cost/benefit of trying to get local source code + dockerized execution vs just doing local development on OSX.

At some point all the issues, setup effort, and operational pain points MAY be resolved well enough, but as of right now my take on this is it's a net loss.

Issue #1: Mounted volumes on Virtual Box (which use vboxfs) are extremely slow

Wait a while and this will almost certainly improve.

Issue #2: File watching is broken

I'm not sure a fix for this is in the near future. If this type of functionality is key to your development workflow, I would consider this a dealbreaker. It's not worth a major R&D effort when compared to just using rbenv/bundler to manage your jekyll/ruby installs and running them locally on OSX like folks have been doing successfully for the past decade+.

Just like "the cloud" has zero involvement in my local development setup, at the moment, docker is a win for testing/staging/deployment and for running databases and other third party components, but the applications I'm actually coding get run straight on OSX.

Bobolink

I second that. We develop on OSX and run the apps directly inside the system (with live reload etc). Then, once the app is complete, we dockerize it for testing, staging and production. – Tipple

Hm, that's a bit of a let down. I've always had parity in my staging/production environments. It's dev that was always the outlier, as I code on OS X. The Docker documentation certainly made it sound like this was a solved problem. I'm going to give it another day of effort and see if I can get something to work. – Cypripedium

Do you still feel this answer as valid today, Peter? Just a few months down the line, but given @Yevgeniy's project and just 2 issues that are now fixed, maybe the cost / benefit is now already worthing! Isn't it? – Joel

It's a personal preference thing. I still wouldn't mess with this because of the sheer quantity of projects I hop between as a consultant. If I was a full-timer working mostly on the same project for weeks/months, it might be worth setting up the rsync/fswatch stuff. – Bobolink

Docker Toolbox is the right way to go about it nowadays because if you use homebrew or another package manager, the docker tool versions will get out of sync, unless they follow the versioning as docker toolbox. – Blomquist

Just a headsup, for now Docker for Mac is even slower than using virtualbox without NFS. forums.docker.com/t/… – Footnote

Answer 3

Docker for Mac and Windows shall be the definitive way of developing with Docker on OS X (and Windows). A Docker product, the software is an “integrated, easy-to-deploy environment for building, assembling, and shipping applications from Mac or Windows.” It purports to be able to solve the issues presented by the OP. From its March 24, 2016 announcement:

• Faster and more reliable: no more VirtualBox! The Docker engine is running in an Alpine Linux distribution on top of an xhyve Virtual Machine on Mac OS X or on a Hyper-V VM on Windows, and that VM is managed by the Docker application. You don’t need docker-machine to run Docker for Mac and Windows.
• Tools integration: Docker for Mac is a Mac application and Docker for Windows is a Windows application, including a native user interface and auto-update capability. The Docker tool set comes bundled with it: Docker command line, Docker Compose, and Docker Notary command line.
• Volume mounting for your code and data: volume data access works correctly, including file change notifications (on Mac inotify now works seamlessly inside containers for volume mounted directories). This enables edit/test cycles for “in container” development.
• Easy access to running containers on the local host network: Docker for Mac and Windows include a DNS server for containers, and are integrated with the Mac OS X and Windows networking system. On a Mac, Docker can be used even when connected to a very restrictive corporate VPN.
• Docker for Mac was architected from scratch to be able to fit the OS X sandbox security model and we are working closely with Apple to achieve this.

Cespitose

I just saw this the other day, and it does look like the most promising solution by far. I'm very excited to give it a shot once it comes out of beta, and if it works well, I'll change it to be the officially accepted answer. – Cypripedium

Unfortunately the current Beta version (1.11.0-beta7) seems to be just as slow as other methods so it might take a while until this is feasible to use forums.docker.com/t/… – Autry

Answer 4

Disclaimer: I might be biased, since I am the author of docker-sync.

I probably tried all the solutions named here, including some more (see the compersion https://github.com/EugenMayer/docker-sync/wiki/Alternatives-to-docker-sync), but they basically either failed on the side of performance (most of them) or on the docker-machine (or none) used / enforced.

http://docker-sync.io has been built to merge all the solutions and provide the best strategies (implementing several, you can choose).

It can be used with rsync (1 way sync) including permission fixes for users, and with unison (2 way sync). It does neither force you into docker-machine or a specific hypervisor, nor requires you to have docker for Mac. It works with all of them.

The performance EugenMayer/docker-sync/wiki/4.-Performance is not influenced, it's like you have no shares at all.

docker-sync and its change-watchers are optimized and do work with projects with 12k files without issues.

Give it a try, if you like, I would love to hear feedback!

Exhibitionism

Answer 5

I feel you! I think I've tried pretty much everything you tried and unfortunately it was still slow. Then I came across this comment https://github.com/boot2docker/boot2docker/issues/64#issuecomment-70689254 that suggests using Vagrant and Parallels and instead of Virtualbox. This allowed me to use nfs and I did indeed saw a big performance boost for my project (Drupal).

Here's the Vagrant file. All you need to do is install vagrant, copy this in a file called Vagrantfile and put it in some folder. Go to that folder and just do a vagrant up instead of your normal boot2docker up.

Vagrant.configure(2) do |config|
  config.vm.box = "parallels/boot2docker"

  config.vm.network "forwarded_port", guest: 80, host: 80

  config.vm.synced_folder(
    "/Users/dicix/work/www", "/vagrant",
    type: 'nfs',
    nfs_udp: true,
    mount_options: %w[actimeo=2],
    bsd__nfs_options: %w[alldirs maproot=root:wheel]
  )
end

Digitiform

I'm assuming this requires a parallels install? – Cypripedium

Answer 6

I'm also using Vagrant with parallels and boot2docker (https://github.com/Parallels/boot2docker-vagrant-box). Development was never easier for me. Works really well with docker-compose and large setups. I don't really feel a delay or massive resource consumption.

This is what my Vagrantfile looks like:

Vagrant.configure(2) do |config|

  config.vm.network "private_network", ip: "192.168.33.10"
  config.vm.box = "parallels/boot2docker"

  config.vm.synced_folder "/Users", "/Users", type: "nfs", mount_options: ["nolock", "vers=3", "udp"], id: "nfs-sync"

end

Bultman

Answer 7

I've been developing in a OS X (mid 2011 Macbook Air) + Boot2Docker + Docker-compose environment for a few weeks now. Haven't run into major performance issues but I avoid running any sort of build when developing (why not use something like jekyll serve --skip-initial-build?). Here's an example docker-compose.yml file I'm using:

docker-compose.yml:

test:
  build: .
  volumes:
    - ./client:/src/client
    - ./server:/src/server
    - ./test:/src/test
  command: nodemon --exec jasmine-node -- test/ --verbose --autotest --captureExceptions --color
  environment:
    - DEBUG=*

Dockerfile:

FROM node:0.12

RUN mkdir -p /src
WORKDIR /src

ENV PATH=/src/node_modules/.bin:$PATH

# We add package.json first so that we the
# image build can use the cache as long as the
# contents of package.json hasn't changed.

COPY package.json /src/
RUN npm install --unsafe-perm

COPY . /src

CMD [ "npm", "start" ]
EXPOSE 3000

I sometimes use NFS (http://syskall.com/using-boot2docker-using-nfs-instead-of-vboxsf/) but haven't noticed a big performance difference when doing so.

For me, the convenience of a simple docker-compose up test to get my environment running has been worth the cost in performance (I routinely work on multiple projects with different stacks).

PS: nodemon is one of the few file watchers which work with vboxsf (see https://github.com/remy/nodemon/issues/419).

Dasha

Even if I skip the initial build with Jekyll, every time I change a file, it will have to rebuild, which still takes on the order of 1-3 minutes if the source code is mounted. This makes it impossible to do any sort of change-and-reload style development. – Cypripedium

@YevgeniyBrikman Oh, I wasn't aware of that :( I guess the last option would be to have your code live inside the boot2docker VM and mount it on your host machine using sshfs. Otherwise, I guess you'll have to wait for better mounted folder performance to use docker as a dev environment. – Dasha

Answer 8

Docker Unison works like a charm! https://github.com/leighmcculloch/docker-unison

Bidirectional Sync with a very good performance!

Nashbar

Answer 9

Getting docker to work as a development tool is possible. But its going to hurt. I've documented the process here :

http://harmingcola.blogspot.com/2015/05/how-to-setup-docker-as-development-tool.html

Kith

Thanks for posting, but how does this solve the performance issues with mounted volumes? – Cypripedium

Ah, sorry, you no longer have to use vBox to mount anything. You can mount folders via the regular docker interface – Kith

Answer 10

This method is the latest (Sep 2015) and easiest way to get Docker setup on Mac: link here:

You install Docker using Docker Toolbox link to instructions here:

It is a complete Docker setup package, that includes the following Docker tools:

Docker Machine for running the docker-machine binary

Docker Engine for running the docker binary

Docker Compose for running the docker-compose binary

Kitematic, the Docker GUI a shell preconfigured for a Docker command-line environment

Oracle VM VirtualBox

What's in the toolbox:

• Docker Client
• Docker Machine
• Docker Compose (Mac only)
• Docker Kitematic
• VirtualBox

Slate

Yes but it unfortunately did not solve the problem originally presented. – Jade