R testthat GitHub Actions snapshot

Question

I'm using GitHub Actions to do the R CMD Check (see https://github.com/r-lib/actions).

I would like to use snapshot_file in testthat script of the sort:

save_png <- function(code, width = 1000, height = 600) {
  path <- tempfile(fileext = ".png")
  grDevices::png(path, width = width, height = height)
  on.exit(dev.off())
  code
  
  path
}

set.seed(123)
df <- data.frame(y=rnorm(20),x=rnorm(20))

test_that("graphs are correct", {
  expect_snapshot_file(path = save_png(plot(df$y,df$x)), name = "plot1.png")
}

At the moment, the error will always be: Adding new file snapshot "_snaps/gets-test1-example/plot1.png". Alternatively, if I upload my plots generated on my laptop, I will get an error that they are different (ever so slightly)...

Because I am aware that snapshot tests are fragile, I thought of two options:

• Delete all snapshot files in the _snaps folder that have been generated e.g. on my laptop and then use some form of upload-artefact (https://github.com/actions/upload-artifact) to upload the snaps the first time - and then GitHub actions has something that is generated by the same machine to test it against (I'm thinking something like a dedicated storage place for each OS, where existing files are first loaded to the Workflow machine and then checked)
• skip all snapshot_files on GitHub Actions (Using skip_on_ci()).

Below is my workflow file, maybe you have an idea how to help me:

on:
  push:
    branches:
      - master
  pull_request:
    branches:
      - master

name: R-CMD-check

jobs:
  R-CMD-check:
    runs-on: ${{ matrix.config.os }}

    name: ${{ matrix.config.os }} (${{ matrix.config.r }})

    strategy:
      fail-fast: false
      matrix:
        config:
          - {os: windows-latest, r: 'release'}
          - {os: macOS-latest, r: 'release'}
          - {os: ubuntu-20.04, r: 'release', rspm: "https://packagemanager.rstudio.com/cran/__linux__/focal/latest"}
          - {os: ubuntu-20.04, r: 'release', rspm: "https://packagemanager.rstudio.com/cran/__linux__/xenial/latest"}
          - {os: ubuntu-20.04, r: 'devel', rspm: "https://packagemanager.rstudio.com/cran/__linux__/focal/latest"}
          - {os: ubuntu-16.04, r: 'release', rspm: "https://packagemanager.rstudio.com/cran/__linux__/xenial/latest"}

    env:
      R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
      RSPM: ${{ matrix.config.rspm }}
    
    defaults:
      run:
        working-directory: gets
    
    steps:
      - uses: actions/checkout@v2

      - uses: r-lib/actions/setup-r@master
        with:
          r-version: ${{ matrix.config.r }}

      - uses: r-lib/actions/setup-pandoc@master

      - name: Query dependencies
        run: |
          install.packages('remotes')
          saveRDS(remotes::dev_package_deps(dependencies = TRUE), "../.github/depends.Rds", version = 2)
          writeLines(sprintf("R-%i.%i", getRversion()$major, getRversion()$minor), "../.github/R-version")
        shell: Rscript {0}

      - name: Cache R packages
        if: runner.os != 'Windows'
        uses: actions/cache@v1
        with:
          path: ${{ env.R_LIBS_USER }}
          key: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-${{ hashFiles('../.github/depends.Rds') }}
          restore-keys: ${{ runner.os }}-${{ hashFiles('../.github/R-version') }}-1-

      - name: Install system dependencies
        if: runner.os == 'Linux'
        run: |
          while read -r cmd
          do
            eval sudo $cmd
          done < <(Rscript -e 'cat(remotes::system_requirements("ubuntu", "20.04","gets"), sep = "\n")')
      - name: Install dependencies
        run: |
          remotes::install_deps(dependencies = TRUE)
          remotes::install_cran("rcmdcheck")
        shell: Rscript {0}

      - name: Check
        env:
          _R_CHECK_CRAN_INCOMING_REMOTE_: false
        run: rcmdcheck::rcmdcheck(args = c("--no-manual", "--as-cran"), error_on = "warning", check_dir = "check")
        shell: Rscript {0}

      - name: Upload check results
        if: failure()
        uses: actions/upload-artifact@main
        with:
          name: ${{ runner.os }}-r${{ matrix.config.r }}-results
          path: check

Saffier

Did you ever find a workable solution for this? I know the snapshot functions are still marked as 'experimental', so maybe the documentation will improve, but at the moment I'm finding it a bit hard to follow. – Evanish

I'm afraid not - I have just continued to run them on my laptop and skip them on ci (GitHub Actions)... – Saffier

Thanks for the question. skip_on_ci() seems to be my best solution, too, which I've learnt from you. – Methedrine

Answer 1

I think your skip suggestion could work if it's a single OS causing issues: use skip_on_os(os = "windows") for example (https://testthat.r-lib.org/reference/skip.html)

You could also use the variant parameter of expect_snapshot_file() to create a snapshot variant per OS.

test_that("graphs are correct", {
  expect_snapshot_file(
    path = save_png(plot(df$y, df$x)),
    name = "plot1.png",
    variant = Sys.info()[["sysname"]]
  )
})

Inherited