Concourse call job from another job with parameters

Question

I have a job with many tasks like this:

- name: main-job
  serial: true
  plan:
  - aggregate:
    - get: <git-resource>
      passed: [previous-job]
      trigger: true
    - get: <git-resource-3>
  - task: <task-1>
    file: <git-resource>/<path>/<task-1-no-db>.yml
  - task: <task-2>
    tags: ['<specific-tag>']
    file: <git-resource>/<path>/<task-1>.yml
    params:
      DATABASE_HOST: <file>
      DATABASE: <my-db-1>
  - task: <task-2>
    tags: ['<specific-tag>']
    file: <git-resource>/<path>/<task-1>.yml
    params:
      DATABASE_HOST: <file>
      DATABASE: <my-db-1>

The problem for me is, I have to literally call the same job but instead of DATABASE params being my-db-1, I want it to be my-db-2.

The only way I am able to do this is by having new job and pass the params, literally copy the entire set of lines. My job is too fat, as in has too many tasks in them, so copying it though is the obvious solution, I am wondering if there's a way to re-use by having multiple pipelines and one main pipeline that essentially calls these pipelines with the param for DATABASE passed or have two small jobs that calls this main job with different params something like this:

- name: <call-main-job-with-db-1>
  serial: true
  plan:
  - aggregate:
    - get: <git-resource>
      passed: [previous-job]
      trigger: true
  - task: <call-main-job-task>
    params:
      DATABASE_HOST: <file>
      DATABASE: <my-db-1>



- name: <call-main-job-with-db-2>
  serial: true
  plan:
  - aggregate:
    - get: <git-resource>
      passed: [previous-job]
      trigger: true
  - task: <call-main-job-task>
    params:
      DATABASE: <my-db-2>

I am not sure if this is even possible since I didn't find any example of this.

Buzzell

Answer 1

You need to just copy and paste the task as you do in the question description. Concourse expects an expressive yaml, there is no branching or logic allowed. If you don't want to copy and paste so much yaml, then you can do some yaml generation magic to simplify what you look at and work with, but concourse will want the full yaml with each job defined separately.

Goodrum

That's what I thought, this answer seems fine for me since it's what concourse expects. – Buzzell

Answer 2

Remember you are using YAML, so you can use YAML features like "Anchors"

You will find some additional information about "Anchors" in this link. Look for "EXTRA YAML FEATURES"

YAML also has a handy feature called 'anchors', which let you easily duplicate content across your document. Both of these keys will have the same value: anchored_content: &anchor_name This string will appear as the value of two keys. other_anchor: *anchor_name

# Anchors can be used to duplicate/inherit properties
base: &base
    name: Everyone has same name

foo: &foo
    <<: *base
    age: 10

bar: &bar
    <<: *base
    age: 20

Try this for your Concourse Pipeline:

common:
  db_common: &db_common
    serial: true
    plan:
    - aggregate:
        - get: <git-resource>
        passed: [previous-job]
        trigger: true
    - task: <call-main-job-task>
        params:

jobs:
- name: <call-main-job-with-db-1>
  <<: *db_common
      DATABASE_HOST: <file>
      DATABASE: <my-db-1>

- name: <call-main-job-with-db-2>
  <<: *db_common
      DATABASE: <my-db-2>

NOTE: Remember that you can have as many Anchors as you want, you can define two or more anchors for the same Job/Task/Resource, etc.

Engrave

Answer 3

Concourse has this fan in fan out paradigm, where you want to keep the jobs simple and short. Use a scripting language e.g. like python or ruby to make your pipeline creation more flexible.

Personally i use one pipeline.yml.erb file where i render different job templates inside. I try to keep my job.yml.erb as generic as possible so i can reuse them for different pipelines.

To bring it to the next level you could specify a meta config.yml and use this config inside your templates to generate your pipeline depending on what you specified in the config.

Wheedle