In this post, we show how a synchronous closure can be converted to an asynchronous closure in Rust. There are several pitfalls we have to work around to make the Rust compiler happy. Unfortunately, the process is not straight forward and hard to tackle for Rust newbies.

Let's start with a very simple program that contains only two functions. A calculator function that executes different mathematical operations and a do_math function that does the actual computation. The operation to execute is a closure, often called lambda function as well.

// Execute multiple mathematical operations by calling a function that takes
// a closure as an argument.
fn calculator() {
    // Closure that increments a value.
    let increment = |x: i64| x + 1;

    // Closure that decrements a value.
    let decrement = |x: i64| x - 1;

    // Run the closures by calling a function that takes the operation (closure) as an argument
    // and a value on which the operation is applied.
    let result = do_math(5, increment);
    assert_eq!(result, 6);
    let result = do_math(5, decrement);
    assert_eq!(result, 4);
}

// Take a closure "operation" and apply it to the value "value".
fn do_math (value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> i64,
{
    // Apply the closure to the value and return the result.
    operation(value)
} 

Of course, we could call the closures directly in calculator instead of the detour through the do_math function, but imagine that the execution of an operation is an exceedingly difficult task and we do not want to write the boilerplate for that every time, so we wrapped that code in the do_math function. For the sake of readability we keep it simple here.

Let's zoom in on two details of the above code.

let increment = |x: i64| x + 1;

The closure is a simple operation that takes one integer of the type i64 and returns the given value incremented by one. The decrement closure works the same.

The second code snipped we need to look at is the function declaration of do_math. It takes an integer of the type i64 and a closure of the type C. But what is C? C is a trait of the type FnOnce that takes an i64 and returns an i64. We don't know what the operation does, only what it takes and what it returns. It can increment a value, decrement it, add 10 to it or do something completely different. If it is a closure that takes an i64 and returns an i64 it will work.

fn do_math (value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> i64 

No async in sight so far. But now imagine that the increment operation takes very long, and we don't want to block the current thread, so we decide to make it async. Prepare yourself for a very unhappy Rust compiler.

Converting the closures to async sounds easy. Add the async keyword where needed, maybe some await here and there, and we are done. If it was that easy, we wouldn't need a blog post explaining it, right? But let's start and convert the closures to async. As a first step we add the async keyword to our closures and check what the compiler says.

Remark: A user on Reddit pointed out that the closures are not async closures, but normal closures that return a future. The Rust feature for async closures is currently unstable in the language. See the tracking issue for the feature for more information.

fn calculator() {
    let increment = |x: i64| async { x + 1 }; // add "async"
    let decrement = |x: i64| async { x - 1 }; // add "async"
    let result = do_math(5, increment);
    assert_eq!(result, 6);
    let result = do_math(5, decrement);
    assert_eq!(result, 4);
}

fn do_math (value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> i64,
{
    operation(value)
}  

If we compile the code above, we get the following complain.

error[E0271]: type mismatch resolving `<[closure@src/async_closures.rs:2:21: 2:45] as FnOnce<(i64,)>>::Output == i64`
  --> src/async_closures.rs:6:18
   |
2  |     let increment = |x: i64| async { x + 1 };
   |                                    --------- the found `async` block
...
6  |     let result = do_math(5, increment);
   |                  ^^^^^^^ expected `i64`, found opaque type
   |
  ::: /home/user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:72:43
   |
72 | pub const fn from_generator (gen: T) -> impl Future |   ------------------------------- the found opaque type
   |
   = note: expected type `i64`
           found opaque type `impl futures::Future `
note: required by a bound in `async_closures::do_math`
  --> src/async_closures.rs:15:23
   |
13 | fn do_math (value: i64, operation: C) -> i64
   |    ------- required by a bound in this
14 | where
15 |     C: FnOnce(i64) -> i64,
   |                       ^^^ required by this bound in `async_closures::do_math`

Oh boy! The error message is longer than our code! Let's dissect what the problem is. The part that helps us clean up the mess is the following:

= note: expected type `i64`
    found opaque type `impl futures::Future ` 

The compiler hints that we changed the type of the closure from |i64| -> i64 to |i64| -> impl futures::Future<Output = i64> by adding the async keyword and now our do_math function can't take the closure as an argument as it still expects a |i64| -> i64 closure (FnOnce(i64) -> i64).

That should be easy to fix. We adjust the type C constraint of the do_math function to return a future. This should to the trick, as that's what the compiler is complaining about.

async fn do_math<C>(value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> Future<Output = i64>,
{
    operation(value)
}

And compile...

error[E0782]: trait objects must include the `dyn` keyword
  --> src/async_closures.rs:17:23
   |
17 |     C: FnOnce(i64) -> Future ,
   |                       ^^^^^^^^^^^^^^^^^^^^
   |
help: add `dyn` keyword before this trait
   |
17 -     C: FnOnce(i64) -> Future ,
17 +     C: FnOnce(i64) -> dyn Future ,
   |
 

That didn't seem to fix it, but instead we got another error. Fortunately, the error looks a lot less intimidating than the error before. The Rust compiler even tells us, how to fix the error. The error E0782 is new in Rust 2021, before that our code would have been allowed. The error was added to indicate that we are working with a trait object instead of a simple heap allocated type. The dyn keyword makes that distinction explicit. So, let's add the dyn keyword to our code and see what happens.

async fn do_math<C>(value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> dyn Future<Output = i64>, // add "dyn"
{
    operation(value)
}

Compiler output:

error[E0271]: type mismatch resolving `<[closure@src/async_closures.rs:4:21: 4:45] as FnOnce<(i64,)>>::Output == (dyn futures::Future + 'static)`
  --> src/async_closures.rs:8:18
   |
4  |     let increment = |x: i64| async { x + 1 };
   |                                    --------- the found `async` block
...
8  |     let result = do_math(5, increment);
   |                  ^^^^^^^ expected trait object `dyn futures::Future`, found opaque type
   |
  ::: /home/user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:72:43
   |
72 | pub const fn from_generator (gen: T) -> impl Future |  ------------------------------- the found opaque type
   |
   = note: expected trait object `(dyn futures::Future + 'static)`
               found opaque type `impl futures::Future `
note: required by a bound in `async_closures::do_math`
  --> src/async_closures.rs:17:23
   |
15 | fn do_math (value: i64, operation: C) -> i64
   |    ------- required by a bound in this
16 | where
17 |     C: FnOnce(i64) -> dyn Future ,
   |                       ^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `async_closures::do_math`
error[E0308]: mismatched types
  --> src/async_closures.rs:19:5
   |
15 | fn do_math (value: i64, operation: C) -> i64
   |                                            --- expected `i64` because of return type
...
19 |     operation(value)
   |     ^^^^^^^^^^^^^^^^ expected `i64`, found trait object `dyn futures::Future`
   |
   = note:      expected type `i64`
           found trait object `(dyn futures::Future + 'static)`

Ok. That got much worse than expected. Let's ignore the first error E0271 for a bit and focus on E0308. The Rust compiler tells us that our types don't match. It expects an i64 but got a dyn futures::Future. In easier words, the compiler would like to have an await, such that the future is unpacked and the content, the i64, is returned. Let's add an await and see what changes. As await can only be used in an async function, we add an async to the function declaration as well.

Making do_math async forces us to make calculator async as well and wait the result from do_math there. The code with all the added async and await keywords looks like this:

use futures::Future;
async fn calculator() {
    let increment = |x: i64| async { x + 1 };
    let decrement = |x: i64| async { x - 1 };
    let result = do_math(5, increment).await;
    assert_eq!(result, 6);
    let result = do_math(5, decrement).await;
    assert_eq!(result, 4);
}
async fn do_math<C>(value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> dyn Future<Output = i64>,
{
    operation(value).await
}

Everything is beautiful asynchronous code now. Let's compile the code again and see the code work.

error[E0271]: type mismatch resolving `<[closure@src/async_closures.rs:4:21: 4:45] as FnOnce<(i64,)>>::Output == (dyn futures::Future<Output = i64> + 'static)`
  --> src/async_closures.rs:8:18
   |
4  |     let increment = |x: i64| async { x + 1 };
   |                                    --------- the found `async` block
...
8  |     let result = do_math(5, increment).await;
   |                  ^^^^^^^ expected trait object `dyn futures::Future`, found opaque type
   |
  ::: /home/user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:72:43
   |
72 | pub const fn from_generator<T>(gen: T) -> impl Future<Output = T::Return>
   |                                           ------------------------------- the found opaque type
   |
   = note: expected trait object `(dyn futures::Future<Output = i64> + 'static)`
               found opaque type `impl futures::Future<Output = i64>`
note: required by a bound in `async_closures::do_math`
  --> src/async_closures.rs:17:23
   |
15 | async fn do_math<C>(value: i64, operation: C) -> i64
   |          ------- required by a bound in this
16 | where
17 |     C: FnOnce(i64) -> dyn Future<Output = i64>,
   |                       ^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `async_closures::do_math`
error[E0277]: the size for values of type `dyn futures::Future<Output = i64>` cannot be known at compilation time
  --> src/async_closures.rs:19:21
   |
19 |     operation(value).await
   |     ----------------^^^^^^ doesn't have a size known at compile-time
   |     |
   |     this call returns `dyn futures::Future<Output = i64>`
   |
   = help: the trait `Sized` is not implemented for `dyn futures::Future<Output = i64>`
   = note: required because of the requirements on the impl of `std::future::IntoFuture` for `dyn futures::Future<Output = i64>`
help: remove the `.await`
   |
19 -     operation(value).await
19 +     operation(value)
   |

Oh common! What's the problem now? Luckily, the compiler error E0277 states what the issue is. It says the trait `Sized` is not implemented for `dyn futures::Future<Output = i64>. Unfortunately, it doesn't tell us how we fix that issue. Do we have to implement the Sized trait for dyn futures::Future<...>? Let's try to understand the issue a bit better.

The Rust compiler needs to know the size of the closures return type, but as it is dyn the size is unknown at compile time and only available at runtime, when a concrete type is handed to the function. One trick that is used a lot in Rust to get around an unknown size at compile time, is to not use the un-sized type directly, but use a pointer on the heap that points to the type. A heap pointer has always a known size. On common 64 bit CPU, the pointer size is 64 bit as well. As we are using Rust, we don't have to handcraft a pointer into some raw memory to point to our type. Instead we can use the Box struct. It takes any value and puts it in a nice box on the heap for us. As Box implements the Deref trait, using it afterwards is nearly transparent. Let's box our return type:

async fn do_math<C>(value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> Box<dyn Future<Output = i64>>, // Add Box
{
    operation(value).await
}

And again, the compiler is unhappy.

error[E0277]: `dyn futures::Future<Output = i64>` cannot be unpinned
  --> src/async_closures.rs:19:21
   |
19 |     operation(value).await
   |     ----------------^^^^^^ the trait `Unpin` is not implemented for `dyn futures::Future<Output = i64>`
   |     |
   |     this call returns `dyn futures::Future<Output = i64>`
   |
   = note: consider using `Box::pin`
   = note: required because of the requirements on the impl of `futures::Future` for `Box<dyn futures::Future<Output = i64>>`
   = note: required because of the requirements on the impl of `std::future::IntoFuture` for `Box<dyn futures::Future<Output = i64>>`
help: remove the `.await`
   |
19 -     operation(value).await
19 +     operation(value)
   |

We hit one of the most difficult to understand topics in Rust: pinned memory. Going to deep into this topic is out of scope for this blog post but have a look at the Rust documentation for Pin if you want to learn more. The brief summary of Pin and Unpin is that pinned memory cannot be relocated. It is fixed at a specific memory location. That is, for example, useful for self-referential structures. Let's fix our code:

use std::pin::Pin;
use futures::Future;
async fn calculator() {
    let increment = |x: i64| async { x + 1 };
    let decrement = |x: i64| async { x - 1 };
    let result = do_math(5, increment).await;
    assert_eq!(result, 6);
    let result = do_math(5, decrement).await;
    assert_eq!(result, 4);
}
async fn do_math<C>(value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> Pin<Box<dyn Future<Output = i64>>>, // Add Pin
{
    operation(value).await
}

Fingers crossed, is the compiler happy now?

error[E0271]: type mismatch resolving `<[closure@src/async_closures.rs:6:21: 6:45] as FnOnce<(i64,)>>::Output == Pin<Box<(dyn futures::Future<Output = i64> + 'static)>>`
  --> src/async_closures.rs:10:18
   |
6  |     let increment = |x: i64| async { x + 1 };
   |                                    --------- the found `async` block
...
10 |     let result = do_math(5, increment).await;
   |                  ^^^^^^^ expected struct `Pin`, found opaque type
   |
  ::: /home/user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:72:43
   |
72 | pub const fn from_generator<T>(gen: T) -> impl Future<Output = T::Return>
   |                                           ------------------------------- the found opaque type
   |
   = note:   expected struct `Pin<Box<(dyn futures::Future<Output = i64> + 'static)>>`
           found opaque type `impl futures::Future<Output = i64>`
note: required by a bound in `async_closures::do_math`
  --> src/async_closures.rs:19:23
   |
17 | async fn do_math<C>(value: i64, operation: C) -> i64
   |          ------- required by a bound in this
18 | where
19 |     C: FnOnce(i64) -> Pin<Box<dyn Future<Output = i64>>>,
   |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `async_closures::do_math`

Again, some types don't match. The Rust compiler knows what it expects as an argument for do_math, but our closures don't meet that requirement. They are neither boxed nor pinned. So, let's fix that.

use std::pin::Pin;
use futures::Future;
async fn calculator() {
    let increment = |x: i64| Box::pin(async move { x + 1 }) as Pin<Box<dyn Future<Output = i64>>>;
    let decrement = |x: i64| Box::pin(async move { x - 1 }) as Pin<Box<dyn Future<Output = i64>>>;
    let result = do_math(5, increment).await;
    assert_eq!(result, 6);
    let result = do_math(5, decrement).await;
    assert_eq!(result, 4);
}
async fn do_math<C>(value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> Pin<Box<dyn Future<Output = i64>>>,
{
    operation(value).await
}

That's it! The compiler is happy and our code works. That was much more painful than it should have been to be honest.

The code we wrote is far from pretty. Luckily there is a crate that helps with cleaning up the mess. The futures crate provides the functionality we coded manually wrapped in nice types. We use the boxed() function to clean up our closures and the BoxFuture type to clean up our do_math function. After applying the syntactic sugar, our program looks like this:

use futures::{future::BoxFuture, FutureExt};
async fn calculator() {
    let increment = |x: i64| async move { x + 1 }.boxed(); // Use boxed()
    let decrement = |x: i64| async move { x - 1 }.boxed(); // Use boxed()
    let result = do_math(5, increment).await;
    assert_eq!(result, 6);
    let result = do_math(5, decrement).await;
    assert_eq!(result, 4);
}
async fn do_math<C>(value: i64, operation: C) -> i64
where
    C: FnOnce(i64) -> BoxFuture<'static, i64>, // Use BoxFuture
{
    operation(value).await
}

That looks much cleaner than before. If you compare this version with our initial synchronous code, the difference isn't that big anymore.

A reader on Reddit pointed out that for this very simple piece of code, there is another solution, which is even cleaner. Keep in mind that in a more complex setup, this may not work.

use std::future::Future;
async fn calculator() {
    let increment = |x: i64| async move { x + 1 };
    let decrement = |x: i64| async move { x - 1 };
    let result = do_math(5, increment).await;
    assert_eq!(result, 6);
    let result = do_math(5, decrement).await;
    assert_eq!(result, 4);
}
async fn do_math<F, C>(value: i64, operation: C) -> i64
where
    F: Future<Output = i64>,
    C: FnOnce(i64) -> F,
{
    operation(value).await
}

As we cannot return a trait directly, we used dyn Trait, which is a type. Another solution is to introduce a generic over a type that is constrained with a trait bound. In our case F is introduced, which is used as the return type from C. F is constrained to implement Future<Output = i64>. This works, as we don't return a trait, but instead a generic type that has to implement that trait.

It is very well possible to write clean asynchronous code with closures in Rust. The problem is that the compiler errors are getting much less helpful as soon as we enter the async-territory. It is our challenge as the Rust community, to improve the async aera of the language, such that new Rust programmers aren't overwhelmed with strange error messages for a seemingly small change.

If you have ideas to improve the code above or found some error, let us know!

All code can from this post can be found on: Github