Gracefully handling executor shutdown.

[Lorak-mmk]

In the project I maintain we have a code like this:
https://github.com/scylladb/scylla-rust-driver/blob/198ca33c00a8949d3bdcc338b44fb03c8c6356bd/scylla/src/client/pager.rs#L988-L999
So we have a Future, with a type-level proof of it sending at least one item to the channel. We spawn this Future, and receive on the channel.
The comment above receiver.recv().await.unwrap() turns out to be wrong, because there are 2 (or 3 if we consider future possible changes to the code) cases where the item may not be sent despite the proof:

• task with this Future panics.
• executor is being shut down.
• (possible in the future) we have a bug in the code and cancel the task ourselves

We can store the JoinHandle of the task, and await it in case None is returned from receiver.recv().await. Then we can propagate the panic.
The issue is what to do with the cases of canceled future. Theoretically we could:

1. Check if executor is being shut down.
2. If not, we have a bug, so panic.
3. If yes, yield all the way up to executor, because we are guaranteed to not continue execution after such yield.

I don't see any nice way to do 1 or 3. For 3, there is yield_now, but it is not guaranteed to yield to executor as there may be some combinators in the stack above it.
Theoretically we could introduce new error variant for this case and return it, but I don't see a point in forcing user to handle an error that only happens when the user doesn't care about it anyway (because the runtime is shutting down).

I'm leaning towards entering a loop with yield_now (and maybe logging error / panicking if we are still in this loop after a few seconds). This seems like extremely ugly solution, is there any nicer way to handle executor shutdown?

[Darksonn]

If you take the solution of awaiting the JoinHandle to get the panic, then that also gives you cancel as the reason. You can handle that by panicking as well, or just return an error.

[Lorak-mmk]

I know I can find out if the task was cancelled. I can't however distinguish if it was cancelled by me, or by executor shutdown.
Why that matters?
If it was cancelled by me, it means there is a bug in the code and I want to panic.
If it was cancelled by executor shutdown, then I want to yield to executor. Panicking is a bug in this case, and returning error makes very little sense. I'd have to add a new error variant for this case, but app is shutting down anyway, so why force user to handle it?. It would be better to somehow yield to executor.

[Darksonn]

Based on your code, the only situation it will be cancelled is runtime shutdown.

[Lorak-mmk]

Yes, we can assume that. Handling other types of cancellation would probably be overly defensive here.
With this assumption, I still don't see how to handle shutdown well, other than entering infinite loop.

[Darksonn]

In principle you don't know that you will be shut down. Maybe the tasks are on different runtimes, and only one was shut down?

If you're going to just sleep until you get cancelled too, then std::future::pending().await would be better than a loop.