Kubernetes can analytical jobs be chained together in a workflow?

Question

Reading the Kubernetes "Run to Completion" documentation, it says that jobs can be run in parallel, but is it possible to chain together a series of jobs that should be run in sequential order (parallel and/or non-parallel).

https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/

---

Or is it up to the user to keep track of which jobs have finished and triggering the next job using a PubSub messaging service?

Should

Check out Argo. – Buccal

Answer 1

Overall, no. Check out things like Airflow for this. Job objects give you a pretty simple way to run a container until it completes, that's about it. The parallelism is in that you can run multiple copies, it's not a full workflow management system :)

Whitefaced

Answer 2

I have used initContainers under the PodSpec in the past to solve problems like this: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox
    command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
  - name: init-mydb
    image: busybox
    command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2; done;']

Take a look here for the chaining of containers using the "depends" keyword is also an option:

https://github.com/kubernetes/kubernetes/issues/1996

Orthoepy

Answer 3

It is not possible to manage job workflows with Kubernetes core API objects.

• Argo Workflow looks like an interesting tool to manage workflow inside Kubernetes: https://argoproj.github.io/projects/argo.
  • It looks like it can handle Kubernetes jobs workflow: https://argoproj.github.io/argo/examples/#kubernetes-resources
  • It is in the CNCF incubator: https://www.cncf.io/blog/2020/04/07/toc-welcomes-argo-into-the-cncf-incubator/

Other alternatives include:

• https://www.nextflow.io/
• https://www.pachyderm.com/
• Airflow: https://airflow.apache.org/docs/apache-airflow/stable/kubernetes.html

This document might also help: https://www.preprints.org/manuscript/202001.0378/v1/download

Rooseveltroost

Turns out that kubeflow pipelines use argo. Stay woke. – Should

Answer 4

Nearly 3 years later, I'll throw another answer into the mix.

Kubeflow Pipelines https://www.kubeflow.org/docs/components/pipelines/overview/pipelines-overview/

Which actually use Argo under the hood.

Should