Function taking an async closure that takes a reference and captures by reference

Question

I want to do something like this:

// NOTE: This doesn't compile

struct A { v: u32 }

async fn foo<
    C: for<'a> FnOnce(&'a A) -> Pin<Box<dyn Future<Output = ()> + 'a>>
>(c: C) {
    c(&A {
        v: 8,
    }).await
}

#[tokio::main]
async fn main() {
    let t = 9;
    foo(|a| async {
        println!("{} {}", t, a.v);
    }.boxed_local()).await;
}

Where a function foo takes an "async closure", gives it a reference, and that "async closure" is also allowed to capture things by reference. Above the compile states t's lifetime needs to be 'static, which makes sense to me.

But I am not sure I understand why when I put a generic lifetime on the type of the reference passed to the "async closure" it compiles:

struct A<'a> { v: u32, _phantom: std::marker::PhantomData<&'a ()> }

async fn foo<
    'b,
    C: for<'a> FnOnce(&'a A<'b>) -> Pin<Box<dyn Future<Output = ()> + 'a>>
>(c: C) {
    c(&A {
        v: 8,
        _phantom: Default::default(),
    }).await
}

#[tokio::main]
async fn main() {
    let t = 9;
    foo(|a| async {
        println!("{} {}", t, a.v);
    }.boxed_local()).await;
}

But then if I add an additional lifetime to A, and foo specifies it as 'static it doesn't compile:

struct A<'a, 'b> { v: u32, _phantom: std::marker::PhantomData<(&'a (), &'b ())> }

async fn foo<
    'b,
    C: for<'a> FnOnce(&'a A<'b, 'static>) -> Pin<Box<dyn Future<Output = ()> + 'a>>
>(c: C) {
    c(&A {
        v: 8,
        _phantom: Default::default(),
    }).await
}

#[tokio::main]
async fn main() {
    let t = 9; // Compile again states t's lifetime needs to be 'static
    foo(|a| async {
        println!("{} {}", t, a.v);
    }.boxed_local()).await;
}

Why does adding the extra lifetime to A, and specifying it as 'static causes t's lifetime to need to longer (i.e. 'static)?

Kahn

Answer 1

TL;DR: This is a limitation of the borrow checker.

---

Before you ask "why didn't it work when I add 'static", you need to ask "why did it work while I didn't have 'static" (TL;DR - implied bounds. You can skip this section if you know what that means).

Let's start from the beginning.

If we have a closure that returns a future, and everything is 'static, everything is fine, of course.

If its returned future needs to depend on its parameters, that's fine, too. Since we're supplying the arguments, we need to tell the compiler "for whatever argument lifetimes we will provide, we want back a future with the same lifetime". You did it, correctly, with HRTB:

type Fut<'a> = Pin<Box<dyn Future<Output = ()> + 'a>>;
async fn foo<C: for<'params> FnOnce(&'params Params) -> Fut<'params>>(c: C)

Now imagine the closure doesn't need its returned future to depend on its arguments, but it does need it to depend on its captured environment. This is possible, too; and since we don't provide the environment (and therefore its lifetimes), and rather it is provided by the creator of the closure - our caller, we need our caller to choose the lifetime. This is easily achievable with a generic lifetime parameter:

async fn foo<'env, C: FnOnce(&Params) -> Fut<'env>>(c: C) {

But what if we need both? This is your case, and it is pretty problematic. The problem is that there is a gap between what you need and what the language lets you express.

What we need (for the parameters, let's ignore the environment for a moment) is "for whatever lifetime I will give, I want a future...".

While what Rust allows you to express with Higher-Ranked Trait Bounds is actually "for whatever lifetime exists, I want...".

Obviously, the problem is that we don't need every lifetime that exists. For instance, "whatever lifetime exists" include 'static. So the closure need to be prepared to be given 'static data and give back a 'static future. But we know we will never give 'static data, yet the compiler is forcing us to handle this impossible case.

There is a potential solution, however. We know we're only ever going to give the closure local variables. The lifetime of local variables will always be shorter than the lifetime of the environment. So, theoretically, we should be able to do:

async fn foo<'env, C: for<'params> FnOnce(&'params Params<'env>) -> Fut<'params>>(c: C) {
    c(&Params { v: 8, _marker: PhantomData }).await
}

Unfortunately, the compiler doesn't agree (yes, I know this compiles, but this is not because the compiler agrees. It disagrees, trust me). It can't conclude that 'env always outlives 'params. And it is right: while it happened to be so, we never guaranteed that. So if the compiler would accept our code based on that, future changes could break customers code accidentally. We went against a core philosohpy of Rust: every potential for breakage must be reflected in the function signature.

How can we reflect the guarantee "we will never give you a lifetime longer than your environment" in the signature? Ah, I've got an idea!

async fn foo<
    'env,
    C:
        for<'params where 'env: 'params>
        FnOnce(&'params Params<'env>) -> Fut<'params>
>(c: C)

Nope. That doesn't work. where clauses are not supported in HRTB (currently; they might in the future).

Or are they?

They aren't supported directly; but there is a way to trick the compiler. There exist implied lifetime bounds.

The idea of implied bounds is simple. Suppose we got the following type:

&'lifetime Type

Here, we know that Type: 'lifetime must hold. That is, every lifetime Type holds must be longer than or equivalent to 'lifetime (more precisely, they are subtypes of 'lifetime, but let's ignore variance here). This is required for &'lifetime Type to be Well-Formed: in simple words, able to exist. If Type contains lifetimes shorter than 'lifetime, and we have a reference for Type with lifetime 'lifetime, we are able to use Type for the whole 'lifetime - even after the shorter lifetimes inside are no longer valid! This can lead to uses-after-free, and because of that we cannot build a reference for a longer lifetime than the lifetime of its referent (you can try).

Since &'lifetime Type can only exist if Type: 'lifetime, and to prevent repetitiveness, if you have &'lifetime Type in your bag (for example, in your argument list), the compiler assumes Type: 'lifetime holds. In other words, having &'lifetime Type implies Type: 'lifetime. And a cruical piece is that these bounds propagate even across for clauses.

If we follow this line of thought, then &'lifetime Type<'other_lifetime> implies 'other_lifetime: 'lifetime (again, ignoring variance). And thus, &'params Params<'env> implies 'env: 'params. Magic! We got our bound without writing it explicitly!

All of this was a necessary background, but it still does not explain why the code fails. The implied bounds should be 'env: 'params and 'static: 'params, both are satisfiable. To understand what happens here, we have to look into the innards of the borrow checker.

---

When the borrow checker sees this closure:

|a| {
    async {
        println!("{} {}", t, a.v);
    }
    .boxed_local()
}

It doesn't anything about it. Specifically, it does not know the lifetimes involved. They are all erased beforehand. The borrow checker does not validate lifetimes of closures - rather, it deduces their requirements and propagate them to the containing function, where they will be validated (and emit errors if they cannot).

The borrow checker sees the following information:

• Closure's type - something like main::{closure#0}.
• Closure's kind - in this case, FnOnce.
• The signature of the call function of the closure. In this case, it is (note that 'env and 'static were erased):

for<'params> extern "rust-call" fn((
    &'params Params<'erased, 'erased>,
)) -> Pin<Box<dyn Future<Output = ()> + 'params>>

• The closure's capture list. In this case, it is &'erased i32 (represented as a tuple, but this is unimpotant). This is a reference to the captured t.

The borrow checker assigns a unique new lifetime for each 'erased lifetime. For simplicity, let's name them 'env and 'my_static for the Params, and 'env_borrow for the t capture.

Now we calculate implied bounds. We have two relevant - 'env: 'params and 'my_static: 'params.

Let's focus on 'env: 'params (more precisely 'env_borrow: 'params. But we can ignore that for our analysis). We cannot prove it ourselves, because 'params is a local lifetime. We declared it ourselves with for<'params>, it did not came from our environment. If we'll gently ask main() to prove 'env: 'params, it'll respond like "'env... hmm, I know 'env, it's the lifetime of the borrow of t. What? 'params? What is that? I don't know it! Sorry, I can't do that for you.". This is not good.

So we want to provide main() with a lifetime it knows. Ho do we do that? Well, we need to find the minimal lifetime that is longer than 'params. This is because if 'env outlives some lifetime bigger than 'params, it definitely outlives 'params itself. We need the minimal lifetime because otherwise it may not provable that 'env: 'some_longer_lifetime even if it is provable that 'env: 'params. There may be several such lifetimes, and we will want to prove them all^[1].

The "bigger" lifetimes in this case are 'env and 'my_static. This is because we have bounds for each, 'env: 'params and 'my_static: 'params (the implied bounds). Thus we know they are bigger (this is not the only constraint, see here for the precise definition).

So we ask main() to prove 'env: 'env (more precisely 'env_borrow: 'env, but again, it doesn't really matter) and 'env: 'my_static. But because my_static is 'static, we will fail to prove that 'env: 'static (again, 'env_borrow: 'static), and therefore we fail, saying that "t does not live long enough".

---

[1] It should be enough to prove only one of them outlives, but per this comment:

// This is slightly too conservative. To show T: '1, given `'2: '1`
// and `'3: '1` we only need to prove that T: '2 *or* T: '3, but to
// avoid potential non-determinism we approximate this by requiring
// T: '1 and T: '2.

I'm not sure what is the non-determinism it is talking about. The PR that introduced this comment is #58347 (specifically commit 79e8c311765) and it says it was done to fix a regression. But it didn't compile even before this PR: even before it we only judged by the constraints we know inside the closure, and we don't know then that 'my_static == 'static. We'd need to propagate the OR bound to the containing function, and to the best of my knowledge this never was the case.

Palomo