What does Apache Mesos do that Kubernetes can't do and vice-versa?

Question

What does Apache Mesos do that Kubernetes can't do or vice-versa?

Mesos is a Two level scheduler. Sure it grabs resource information from every machine and gives it to the top level scheduler such that frameworks like kubernetes can use to schedule containers across machines but Kubernetes can itself schedule containers across machines (No need for Mesos from this regard). so what are few things that Apache Mesos can do that Kubernetes cannot do or vice-versa?

Dickens

Have you checked this question? #26705701 – Remonstrance

I'd add that Kubernetes does in fact support multiple schedulers out of the box. – Ullund

Answer 1

Both Mesos and Kubernetes are n-th level containers orchestrators. This means you can achieve the same features but some kind of tasks could be done easier (read. better) on one of them. In fact, you can run Kubernetes on Mesos and vice verse.

Let's go through main differences that give some clue when you need to make a decision:

Architecture

As you pointed out Mesos is a Two-Level Scheduler and this is the main difference in architecture. This gives you the ability to create your custom scheduler (aka framework) to run your tasks. What's more, you can have more than one scheduler. All your schedulers compete for the resources that are fairly distributed using Dominant Resources Fairness algorithm (that could be replaced with custom allocator). You can also assign roles to the frameworks and tasks and assign weights to this roles to prioritize some schedulers. Roles are tightly connected with resources. Above features gives you the ability to create your own way of scheduling for different applications (e.g., Fenzo) with different heuristics based on a type of tasks you want to run. For example, when running batch tasks it's good to place them near data and time to start is not so important. On the other hand, running stateless services is independent of nodes and it's more critical to run them ASAP.

Kubernetes architecture is a single level scheduler. That means decisions where pod will be run are made in a single component. There is no such thing as resource offer. On the other hand, everything there is pluggable and built with a layered design.

Origin

Mesos was created at Twitter (formerly at Berkeley but the first production usage was at Twitter) to support their scale.

In March 2010, about a year into the Mesos project, Hindman and his Berkeley colleagues gave a talk at Twitter. At first, he was disappointed. Only about eight people showed up. But then Twitter's chief scientist told him that eight people was lot – about ten percent of the company's entire staff. And then, after the talk, three of those people approached him.

Soon, Hindman was consulting at Twitter, working hand-in-hand with those ex-Google engineers and others to expand the project. Then he joined the company as an intern. And, a year after that, he signed on as a full-time employee. source

Kubernetes was created by Google to bring users to their cloud promising no lock-in experience. This is the same technique Amazon did with Kindle. You can read any book on it but using it with Amazon gives you the best experience. The same is true for Google. You can run Kubernetes on any cloud (public or private) but the best tooling, integration and support you'll get only on Google Cloud.

But Google and Microsoft are different. Microsoft wants to support everything on Azure, while Google wants Kubernetes everywhere. (In a sense, Microsoft is living up to the Borg name, assimilating all orchestrators, more than Google is.) And quite literally, Kubernetes is how Google is playing up to the on-premises cloud crowd giving it differentiation from AWS (which won’t sell its infrastructure as a stack with a license, although it says VMware is its private cloud partner) and Microsoft (which still doesn’t have its Azure Stack private cloud out the door). source

Community

Judging a project simply by its community size could be misleading. It's like you'd be saying that php is a great language because it has large community.

Mesos community is much smaller than Kubernetes. That's the fact. Kubernetes has financial support from many big companies including Google, Intel, Mirantis, RedHat and more while Mesos is developed mainly by Mesosphere with some support from Apple, Microsoft. Although Mesos is a mature project, its development is slow but stable. On the other hand, Kubernetes is much younger, but rapidly developed.

Meso contributors origin

The Kubernetes Community - Ian Lewis, Developer Advocate, Google

Scale

Mesos was targeted for big customers from the early beginning. It is used at Twitter, Apple, Verizon, Yelp, Netflix to run hundreds of thousands of containers on thousands of servers.

Kubernetes was started by Google to give developers Google Infrastructure experience (GIFFE). From the beginning, it was prepared for small scale up to hundreds of machines. This constraint is increased with every release but they started small to grow big. There are no public data about biggest Kubernetes installation.

Hype

Due to scale issues, Kuberntetes started to be popular among smaller companies (not cloud scale) while Mesos was targeted for enterprise users. Kubernetes is supported by Cloud Native Foundation while Mesos is Apache Foundation Project. These two foundations have different founding and sponsors. Generally, more money gives you better marketing and Kubernetes definitely did it right.

https://g.co/trends/RUuhA

Conclusion

It looks like Kubernetes already won the containers orchestrator war. But if you have some custom workloads and really big scale, Mesos could be a good choice.

The main difference is in the community size and the open source model : where DCOS is supported by Mesosphere and provide enterprise features in a commercial product only (because mesosphere isn't philanthropist), K8S has a larger community with strong contributions from different companies resulting in providing much more integrated enterprise features (multitenancy, RBAC, quota, preemption, gateways...) meaning they are easier to use, not necessarily they don't exist in DCOS. I would globally say that :

• DCOS is more battle tested for stateful and big data workloads but lacks of integration with other perimetric components including plug and play central monitoring and logging and enterprise features like security model, multi tenancy, auto updates... It was a very hard way to integrate everything for a production grade platform.
• K8S is more battle tested for stateless apps and provides lots of plug and play tools like prometheus, EFK, helm... which makes the implementation of a production grade platform much easier. Next to that there is a big move on stateful workloads with statefulsets and the operator pattern which is comparable with mesos frameworks but again, K8S provides lots of tools to develop them with less costs because lots of functionalities are provided out of the box, it takes me 2 months to develop a MongoDB operator to provide MongoDB as a service in a multi tenant and secured way and I needed to learn Golang in the same time.

source

• https://www.infoworld.com/article/3118345/cloud-computing/why-kubernetes-is-winning-the-container-war.html
• https://www.theregister.co.uk/2017/10/17/docker_ee_kubernetes_support
• https://www.techrepublic.com/article/these-two-vendors-are-most-likely-to-bring-kubernetes-containers-to-the-enterprise
• https://www.cloudhealthtech.com/blog/container-wars-are-over-kubernetes-has-won
• https://news.ycombinator.com/item?id=12462261

Slipon

Can you please talk about advantages and disadvantages of two level scheduler vs single level scheduler on how they would perform on small and large clusters? – Dickens

Short answer: it depends :) Aurora supports Twitter scale and a single master is capable of taking care of big installation. In theory, 2-level scheduler will always be less efficient than single-level because of communication with frameworks. The bottleneck of this design is communication which does not happen in a single layer. Another example is Google Omega which revers the way frameworks claims resources which are not offered but queried. – Slipon

I actually hear that single level schedulers are not so good at making optimal decisions on large clusters where two level schedulers seem to excel. is that true? – Dickens

Again it depends. E.g. with Mesos you can use different schedule algorithm for different workloads. If you have a single framework on top of Mesos you actually have the single layer scheduler. – Slipon

I would like to add that AWS is providing a hosted Kubernettes and soon bare metal Docker containers, so AWS offering it will be similar to GCP, although I agree that GCP has the best integration with Kubernettes. – Finery

It sounds to me that winning "containers orchestrator war" is all about who can supply more free pizza and a beer! – Dickens

I agree. It's not about who is running bigger clusters but who is getting more buzz about it – Slipon

@Slipon Very naive question from a beginner: What is the reason that warrants a separate scheduler? Scheduler sounds like a very tiny part in comparison to the Resource/Cluster Manager. If available resources can be found, then Mesos itself can just pass the jobs/tasks instead of Marathon/Aurora? What are all the works that is being done by a scheduler at a fine grained/granular level to really appreciate the need for a separate scheduler ? – Nonrepresentational

Good quesition. I hope I understand it right. The ideda behind separating schedulers is to allow making indepenedent decision on how task should be launched. Scheduler can prioritize jobs, take care of redeployment, handle split brain situation. This is really hard to handle in one big application. E.g., batch jobs failures will be handled differently from long running tasks. – Slipon

do not know what you means the two level vs one layer, both are n-th level, what is the difference between level and layer? – Longwood

@Longwood I updated the answer. Pleas read closely both are n-th level orchestrator and Mesos is two level scheduler while kuberentes is single level scheduler. – Slipon