How to test for multiple warnings in an unknown order using testthat?

Question

I want to test that a function generates multiple warnings (4 or more), when the order of the warnings can vary. My best attempt to do this is based on look-ahead RegExp matching. Simplifying to just 2 warnings, I know my RegExp work on a single string output, because both of the following are true:

grepl("(?s)(?=.*2)(?=.*1)", "* warn 1.\n* warn 2.", perl=TRUE)
grepl("(?s)(?=.*2)(?=.*1)", "* warn 2.\n* warn 1.", perl=TRUE)

However, this does not work when testing multiple warnings with testhat::expect_warning

# The function generating warnings:
foo <- function() { warning("warn 1."); warning("warn 2.") }
foo()
Warning messages:
1: In foo() : warn 1.
2: In foo() : warn 2.

# Testing it
expect_warning( foo(), "(?s)(?=.*1)(?=.*2)", perl=TRUE)

Error: foo() does not match '(?s)(?=.*1)(?=.*2)'. Actual values:
* warn 1.
* warn 2.

I suspect this is because the internals of expect_warning are doing something like testing the given RegExp against each warning separately--why an expect_warning( ... all=TRUE ) parameter might be meaningful.

Unfortunately I can't use this with a RegExp like "1 | 2"; that succeeds if only one warning is given.

I also want to avoid running the function multiple times and testing a different warning each time. There is a large amount of setup and teardown code needed to test the real function. It interacts heavily with the file system, and since it is the file-system warnings I am testing for, I can't mock that. Moreover, I want to test for the warnings in multiple situations, each of which requires different setup and teardown code, so this quickly bloats my tests.

Any suggestion on how to test for multiple warnings simply and all at once?

Blastema

It seems that expect_warning has no perl parameter (as other functions in R), but why not simply use the pattern 1.*\\n.*2 or 1(.*\\n)*.*2 (Since (?s), .* and the lookaheads are useless)? – Holo

@Casimir et Hippolyte - expect_warning takes ..., and passes parameters through to the underlying grepl call, so it will use the perl= TRUE parameter. Without it a "lookahead" containing RegExp is invalid. I need that because the order is not known, and there are more than 2 in my real application. The lookaheads are a way to match elements in an unknown order. – Blastema

Answer 1

So I have cobbled together a solution that involves using my own warning catch loop and checking the warnings as strings. Not an "all at once" solution, but it is at least compact given a complex function call. My code as it would apply to the example function foo() is the following:

gotWarnings= character(0)
withCallingHandlers({
   got <- foo()
   }, warning= function(e) {
      # Push warning onto vector in parent frame.
      gotWarnings <<- c(gotWarnings, conditionMessage(e))
      invokeRestart("muffleWarning")
})

# Ensure no unexpected warnings, 
expect_equal(length(gotWarnings), 2)

# Test that each warning I want is there
expect_true( any( grepl( "warn 1\\.", gotWarnings )))
expect_true( any( grepl( "warn 2\\.", gotWarnings )))

Figuring out how to continue processing after catching a warning was the hard part; tryCatch exits after catching the first warning. Here and here helped me work this out. Perhaps this could be converted into an expect_all_warnings test that takes a vector of RegEx to match, maybe with all= TRUE meaning no warning can go unmatched. I'll hold off accepting this as an answer in case somebody posts a better one, or at least one that packages a solution like this nicely.

Blastema

Feels kind of weird accepting my own answer as a solution, but it's been a while, and someone up-voted the question. – Blastema

I have a similar problem when testing complete scripts. I expect several errors but I do not want to execute the script for each test. Is there a way to capture all warnings and then test them? I wrote a SO question already – Pitsaw

Answer 2

To capture the warnings and analyze them "by hand", you can also use testthat::capture_warnings:

# The function generating warnings:
foo <- function() { warning("warn 1."); warning("warn 2.") }

w <- capture_warnings(foo())
expect_match(w, ".*1", all = FALSE)
expect_match(w, ".*2", all = FALSE)
expect_match(w, ".*3", all = FALSE)

(The last line raises an error.)

Fushih

Answer 3

You can maybe use the [12] like:

expect_warning( foo(),'(?s)(?=.*[12])' , all=T, perl=TRUE)

Velasquez

Nope. Same problem as 1 | 2. True if only one warning given, for example it is true for a function bar <- function() { warning("warn 1.") } – Blastema