What is the output of loop task in argo?

Question

As per the Argo DAG template documentation.

tasks.<TASKNAME>.outputs.parameters: When the previous task uses 'withItems' or 'withParams', this contains a JSON array of the output parameter maps of each invocation

When trying with the following simple workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-workflow-
spec:
  entrypoint: start
  templates:
  - name: start
    dag:
      tasks:
        - name: with-items
          template: hello-letter
          arguments:
            parameters:
            - name: input-letter
              value: "{{item}}"
          withItems:
          - A
          - B
          - C
        - name: show-result
          dependencies:
          - with-items
          template: echo-result
          arguments:
            parameters:
            - name: input
              value: "{{tasks.with-items.outputs.parameters}}"

  - name: hello-letter
    inputs:
      parameters:
      - name: input-letter
    outputs:
      parameters:
      - name: output-letter
        value: "{{inputs.parameters.input-letter}}"
    script:
      image: alpine
      command: ["sh"]
      source: |
        echo "{{inputs.parameters.input-letter}}"

  - name: echo-result
    inputs:
      parameters:
      - name: input
    outputs:
      parameters:
      - name: output
        value: "{{inputs.parameters.input}}"
    script:
      image: alpine
      command: ["sh"]
      source: |
        echo {{inputs.parameters.input}}

I get the following error : Failed to submit workflow: templates.start.tasks.show-result failed to resolve {{tasks.with-items.outputs.parameters}}

Argo version (running in a minikube cluster)

argo: v2.10.0+195c6d8.dirty
  BuildDate: 2020-08-18T23:06:32Z
  GitCommit: 195c6d8310a70b07043b9df5c988d5a62dafe00d
  GitTreeState: dirty
  GitTag: v2.10.0
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

Same error in Argo 2.8.1, although, using .result instead of .parameters in the show-result task worked fine (result was [A,B,C]), but doesn't work in 2.10 anymore

        - name: show-result
          dependencies:
          - with-items
          template: echo-result
          arguments:
            parameters:
            - name: input
              value: "{{tasks.with-items.outputs.result}}"

The result:

STEP                                TEMPLATE      PODNAME                                     DURATION  MESSAGE
 ⚠ test-workflow-parallelism-xngg4  start                                                                                                                     
 ├-✔ with-items(0:A)                hello-letter  test-workflow-parallelism-xngg4-3307649634  6s                                                              
 ├-✔ with-items(1:B)                hello-letter  test-workflow-parallelism-xngg4-768315880   7s                                                              
 ├-✔ with-items(2:C)                hello-letter  test-workflow-parallelism-xngg4-2631126026  9s                                                              
 └-⚠ show-result                    echo-result                                                         invalid character 'A' looking for beginning of value

I also tried to change the show-result task as:

        - name: show-result
          dependencies:
          - with-items
          template: echo-result
          arguments:
            parameters:
            - name: input
              value: "{{tasks.with-items.outputs.parameters.output-letter}}"

Executes without no errors:

STEP                                TEMPLATE      PODNAME                                     DURATION  MESSAGE
 ✔ test-workflow-parallelism-qvp72  start                                                                 
 ├-✔ with-items(0:A)                hello-letter  test-workflow-parallelism-qvp72-4221274474  8s          
 ├-✔ with-items(1:B)                hello-letter  test-workflow-parallelism-qvp72-112866000   9s          
 ├-✔ with-items(2:C)                hello-letter  test-workflow-parallelism-qvp72-1975676146  6s          
 └-✔ show-result                    echo-result   test-workflow-parallelism-qvp72-3460867848  3s 

But the parameter is not replaced by the value:

argo logs test-workflow-parallelism-qvp72
test-workflow-parallelism-qvp72-1975676146: 2020-08-25T14:52:50.622496755Z C
test-workflow-parallelism-qvp72-4221274474: 2020-08-25T14:52:52.228602517Z A
test-workflow-parallelism-qvp72-112866000: 2020-08-25T14:52:53.664320195Z B
test-workflow-parallelism-qvp72-3460867848: 2020-08-25T14:52:59.628892135Z {{tasks.with-items.outputs.parameters.output-letter}}

I don't understand what to expect as the output of a loop! What did I miss? Is there a way to find out what's happening?

Clad

Any progress on this? I am running into the same problem. – Suzannesuzerain

I faced the same issue with artifacts as well. While I could not solve it directly with the input-output, I solved with volume mounts. In my case, the templates running in a loop were producing some files that I needed to collate in one place. For this, I used a volumeClaimTemplates and mounted it in the loop pods and stored the files there. Then I could access it in the collate-pod directly. – Jugum

Answer 1

This is quite a common problem I've faced, I have not come across this in any bug report or feature documentation so far so it's yet to be determined if this is a feature or bug. However argo is clearly not capable in performing a "map-reduce" flow OOB.

The only "real" workaround I've found is to attach an artifact, write the with-items task output to it, and pass it along to your next step where you'll do the "reduce" yourself in code/script by reading values from the artifact.

---- edit -----

As mentioned by another answer this was indeed a bug which was resolved for the latest version, this resolves the usage of parameters as you mentioned as an option but outputs.result still causes an error post bugfix.

Belt

Found the bug. :-) Linked in my answer. – Malnourished

Answer 2

There was a bug which caused this error before Argo version 3.2.5. Upgrade to latest and try again.

It looks like the problem was in the CLI only. I submitted the workflow with kubectl apply, and it ran fine. The error only appeared with argo submit.

The argo submit error was resolved when I upgraded to 3.2.6.

Malnourished

Answer 3

This issue is currently open on Argo Workflows Github: issue #6805

You could use a nested DAG to workaround this issue. This helps the parallel executions artifact resolution problem because each task output artifact is scoped to its inner nested DAG only, so there's only one upstream branch in the dependency tree. The error in issue #6805 happens when artifacts exist in the previous step and there's more than one upstream branch in the dependency tree.

Hardman