Github actions: merge artifacts after matrix steps
Asked Answered
Z

1

6

I have a job in Github Actions workflow that runs unit tests and then uploads reports to Jira Xray. The thing is tests step takes quite a while to complete, so I want to split task execution into a few smaller chunks using matrix.

I did it for linting and it works well, however for unit tests I'm struggling with how can I collect and merge all reports together so they can be uploaded after all matrix steps are done.

Here's how current unit tests step looks like

 unit-test:
    runs-on: ubuntu-latest
    needs: setup
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: actions/cache@v3
        with:
          path: ${{ env.CACHE_NODE_MODULES_PATH }}
          key: build-${{ hashFiles('**/package-lock.json') }}
      - run: npx nx affected:test --parallel=3 --base=${{ env.BASE_REF}} --head=HEAD # actual unit tests
      - name: Check file existence #checking whether there're reports at all
        if: success() || failure()
        id: check_files
        uses: andstor/file-existence-action@v1
        with:
      # all reports will be placed in this directory
      # for matrix job reports will be separated between agents, so it's required to merge them
          files: 'reports/**/test-*.xml' 
      - name: Import results to Xray
        if: (success() || failure()) && steps.check_files.outputs.files_exists == 'true' && github.event_name == 'push'
        uses: mikepenz/xray-action@v2
        with:
          username: ${{ secrets.XRAY_CLIENT_ID }}
          password: ${{ secrets.XRAY_CLIENT_SECRET }}
          testFormat: 'junit'
          testPaths: 'reports/**/test-*.xml' # that's where I need to grab all reports
          projectKey: 'MY_KEY'
          combineInSingleTestExec: true

Matrix job for linting looks like this. I would like to do the same for unit tests, but at the same time I want to collect all reports as it works in the job above

linting:
    runs-on: ubuntu-latest
    needs: [setup]
    strategy:
      matrix:
        step: ${{ fromJson(needs.setup.outputs.lint-bins) }} # this will be something like [1,2,3,4]
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: actions/cache@v3
        with:
          path: ${{ env.CACHE_NODE_MODULES_PATH }}
          key: build-${{ hashFiles('**/package-lock.json') }}
      # some nodejs logic to run few jobs, it uses "execSync" from "child_process" to invoke the task
      - run: node scripts/ci-run-many.mjs --target=lint --outputTarget=execute --partNumber=${{ matrix.step }} --base=${{ env.BASE_REF}} --head=HEAD 
Zooplankton answered 26/8, 2022 at 8:31 Comment(0)
Z
9

Figured it myself

  unit-test:
    runs-on: ubuntu-latest
    needs: [setup]
    strategy:
      fail-fast: false
      matrix:
        step: ${{ fromJson(needs.setup.outputs.unit-test-bins) }}
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: actions/cache@v3
        with:
          path: ${{ env.CACHE_NODE_MODULES_PATH }}
          key: build-${{ hashFiles('**/package-lock.json') }}
      - run: node scripts/ci-run-many.mjs --target=test --outputTarget=execute --partNumber=${{ matrix.step }} --base=${{ env.BASE_REF}} --head=HEAD
      - name: Upload reports' artifacts 
        if: success() || failure()
        uses: actions/upload-artifact@v3
        with:
          name: ${{ env.RUN_UNIQUE_ID }}_artifact_${{ matrix.step }}
          if-no-files-found: ignore
          path: reports
          retention-days: 1

  process-test-data:
    runs-on: ubuntu-latest
    needs: unit-test
    if: success() || failure()
    steps:
      - uses: actions/checkout@v3

      - name: Download reports' artifacts
        uses: actions/download-artifact@v3
        with:
          path: downloaded_artifacts
          
      - name: Place reports' artifacts
        run: rsync -av downloaded_artifacts/*/*/ unit_test_reports/
      - name: Check reports existence
        id: check_files
        uses: andstor/file-existence-action@v1
        with:
          files: 'unit_test_reports/**/test-*.xml'
      - name: Import results to Xray
        run: ls -R unit_test_reports/
Zooplankton answered 1/9, 2022 at 11:24 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.