Marathon vs Kubernetes vs Docker Swarm on DC/OS with Docker containers

Question

I'm looking for some pros and cons of whether to go with Marathon and Chronos, Docker Swarm or Kubernetes when running Docker containers on DC/OS.

For example, when is it better to use Marathon/Chronos than Kubernetes and vice versa?

Right now I'm mostly into experimenting but hopefully we'll start using one of these services in production after the summer. This may disqualify Docker Swarm since I'm not sure if it'll be production ready by then.

What I like about Docker Swarm is that it's essentially just "Docker commands" and you don't have to learn something completely new. We're already using docker-compose and that will work out of the box with Docker Swarm (at least in theory) so that would be a big plus. My main concern with Docker Swarm is if it'll cover all use cases required to run a system in production.

Masorete

Answer 1

I'll try to break down the unique aspects of each container orchestration framework on Mesos.

Use Docker Swarm if:

• You want to use the familiar Docker API to launch Docker containers on Mesos.
• Swarm may eventually provide an API to talk to Kubernetes (even K8s-Mesos) too.
• See: http://www.techrepublic.com/article/docker-and-mesos-like-peanut-butter-and-jelly/

Use Kubernetes-Mesos if:

• You want to launch K8s Pods, which are groups of containers co-scheduled and co-located together, sharing resources.
• You want to launch a service alongside one or more sidekick containers (e.g. log archiver, metrics monitor) that live next to the parent container.
• You want to use the K8s label-based service-discovery, load-balancing, and replication control.
• See http://kubernetesio.blogspot.com/2015/04/kubernetes-and-mesosphere-dcos.html

Use Marathon if:

• You want to launch Docker or non-Docker long-running apps/services.
• You want to use Mesos attributes for constraint-based scheduling.
• You want to use Application Groups and Dependencies to launch, scale, or upgrade related services.
• You want to use health checks to automatically restart unhealthy services or rollback unhealthy deployments/upgrades.
• You want to integrate HAProxy or Consul for service discovery.
• You want to launch and monitor apps through a web UI or REST API.
• You want to use a framework built from the start with Mesos in mind.

Use Chronos if:

• You want to launch Docker or non-Docker tasks that are expected to exit.
• You want to schedule a task to run at a specific time/schedule (a la cron).
• You want to schedule a DAG workflow of dependent tasks.
• You want to launch and monitor jobs through a web UI or REST API.
• You want to use a framework built from the start with Mesos in mind.

Gosselin

I just wanted to add that as of K8s 1.6 it to supports the following (some of it for a long time): * Docker-CRI (beta) and cri-o, frakti, rkt (alpha) for non Docker containers. * Health checks to see when a container has started/is no longer responding. * Recreation of unhealthy pods. * Cron like jobs, both recurring and once. * Batch jobs (started manually and runs to completion once). Since Mesosphere themselves say K8s is a first class citizen on Mesos the "built from the start" argument feels a bit wague as well... – Dene

Answer 2

Though it's a bit outdated, it may be helpful to read What's the difference between Apache's Mesos and Google's Kubernetes, to get some of the basics right. Also, note that Mesos operates on a different level than Kubernetes/Marathon/Chronos. Last but not least, see Docker Swarm + Mesos by Timothy Chen, keeping in mind that Marathon and Swarm can operate simultaneously on the same Mesos cluster.

Fritter