Is there a simple way to conditionally enable or ignore entire test suites in Rust?

Question

I’m working on a Rust library that provides access to some hardware devices. There are two device types, 1 and 2, and the functionality for type 2 is a superset of the functionality for type 1.

I want to provide different test suites for different circumstances:

• tests with no connected device (basic sanity checks, e. g. for CI servers)
• tests for the shared functionality (requires a device of type 1 or 2)
• tests for the type 2 exclusive functionality (requires a device of type 2)

I’m using features to represent this behavior: a default feature test-no-device and optional features test-type-one and test-type-two. Then I use the cfg_attr attribute to ignore the tests based on the selected features:

#[test]
#[cfg_attr(not(feature = "test-type-two"), ignore)]
fn test_exclusive() {
    // ...
}

#[test]
#[cfg_attr(not(any(feature = "test-type-two", feature = "test-type-one")), ignore)]
fn test_shared() {
    // ...
}

This is rather cumbersome as I have to duplicate this condition for every test and the conditions are hard to read and maintain.

Is there any simpler way to manage the test suites?

I tried to set the ignore attribute when declaring the module, but apparently it can only be set for each test function. I think I could disable compilation of the excluded tests by using cfg on the module, but as the tests should always compile, I would like to avoid that.

Benzoin

You could try to annotate a complete mod block instead of single functions – Sandler

@Sandler But this only works if I want to disable compiling, not for ignoring the test cases, right? – Benzoin

Answer 1

Is there a simple way to conditionally enable or ignore entire test suites in Rust?

The easiest is to not even compile the tests:

#[cfg(test)]
mod test {
    #[test]
    fn no_device_needed() {}

    #[cfg(feature = "test1")]
    mod test1 {
        fn device_one_needed() {}
    }

    #[cfg(feature = "test2")]
    mod test2 {
        fn device_two_needed() {}
    }
}

I have to duplicate this condition for every test and the conditions are hard to read and maintain.

1. Can you represent the desired functionality in pure Rust? yes
2. Is the existing syntax overly verbose? yes

This is a candidate for a macro.

macro_rules! device_test {
    (no-device, $name:ident, {$($body:tt)+}) => (
        #[test]
        fn $name() {
            $($body)+
        }
    );
    (device1, $name:ident, {$($body:tt)+}) => (
        #[test]
        #[cfg_attr(not(feature = "test-type-one"), ignore)]
        fn $name() {
            $($body)+
        }
    );
    (device2, $name:ident, {$($body:tt)+}) => (
        #[test]
        #[cfg_attr(not(feature = "test-type-two"), ignore)]
        fn $name() {
            $($body)+
        }
    );
}

device_test!(no-device, one, {
    assert_eq!(2, 1+1)
});

device_test!(device1, two, {
    assert_eq!(3, 1+1)
});

the functionality for type 2 is a superset of the functionality for type 1

Reflect that in your feature definitions to simplify the code:

[features]
test1 = []
test2 = ["test1"]

If you do this, you shouldn't need to have any or all in your config attributes.

a default feature test-no-device

This doesn't seem useful; instead use normal tests guarded by the normal test config:

#[cfg(test)]
mod test {
    #[test]
    fn no_device_needed() {}
}

If you follow this, you can remove this case from the macro.

---

I think if you follow both suggestions, you don't even need the macro.

Oleo

Answer 2

It's not a perfect solution, but i had a similar issue (slow database tests that I didn't want to run as part of my regular unit tests.) My solution was to mark every test in the test suite [ignore] and then use cargo test suite-name -- --include--ignored:

// In my_module.rs
#[cfg(test)]
mod test {
    #[ignore]
    #[test]
    fn test_one () {
       ...
    }

    #[ignore]
    #[test]
    fn test_two () {
       ...
    }

    ...
}

These test compiled by cargo test but not run. Then you can specifically run these tests with cargo test my_module::test -- --include-ignored.

Its annoying to need to [ignore] each test individually but at least it gives you a way to run that test suite specifically without running other ignored tests from other modules.

Nodarse