How to know if all the receivers has been dropped?

[dvaerum]

How to know if all the receivers has been dropped? now that send function no longer returns anything.

crossbeam-channel/src/internal/channel.rs

Lines 299 to 305 in 5050c89

	pub fn send(&self, msg: T) {
	match &self.0.flavor {
	ChannelFlavor::Array(chan) => chan.send(msg),
	ChannelFlavor::List(chan) => chan.send(msg),
	ChannelFlavor::Zero(chan) => chan.send(msg),
	}
	}

Dropping receivers doesn't close the channel anymore, but one can still relatively easily implement the old interface where they do. See this for an example:

https://github.com/crossbeam-rs/crossbeam-channel/blob/master/examples/mpsc.rs

[Flaise]

That's not a trivial amount of work you linked to and I have this use case too - I'm writing a program with one main thread that reads from a receiver with several worker threads that send messages to it and exit if the main thread terminates and drops its receiver at any time while they're blocking on std/mpsc send operations. I'd like to use crossbeam_channel instead because it's a better library... except for this one feature that I'd have to reimplement myself. (Aborting the program when the main thread exits wouldn't be enough because my unit tests would block without this behavior.)

Since we have more than one use case for this feature, it looks to me like it would be better in this library than in call sites of this library.

@Flaise How about the following solution?

// Spawns a thread and returns a channel that signals its termination.
fn spawn(f: impl FnOnce()) -> Receiver<()> {
    let (s, r) = channel::bounded(0);
    thread::spawn(move || {
        f();
        drop(s);
    });
    r
}

// The channel for sending messages.
let (s, r) = channel::unbounded();

// Create the main thread.
let done = spawn(move || {
    loop {
        let msg = r.recv();
        // Do something...
    }
});

// Creater worker threads.
for _ in 0..WORKER {
    let s = s.clone();
    let done = done.clone();
    spawn(move || {
        loop {
            let msg = produce_message();
            select! {
                send(s, msg) => {}
                recv(done) => break
            }
        }
    });
}

[BusyJay]

I think this is a very common situation that needs to be detected. And if all receivers are dropped, and sender keeps sending messages, the program may OOM, which is not obvious to tell and debug.

Duplicate issue: #55

Pinging people who might be interested: @vorner @jdm @SimonSapin @BurntSushi @glaebhoerl @danburkert @matthieu-m @coder543

Here's a suggestion on how we might solve this issue.

Several people already have expressed interest in being able to detect whether all receivers have been dropped during send.

Currently, send will just send a message into the channel without caring about dropped receivers, which might lead to deadlocks and memory leaks. This API decision was not made lightly and has faced controversy - see #39 and BurntSushi/chan#2.

If we change the behavior of send so that it returns a Result indicating whether receivers have been dropped, then it's not clear how should send(s, msg) inside select! behave, which leads us down another rabbit hole... It's a long story.

Instead, I propose we simply add a new method to Sender:

impl<T> Sender<T> {
    // Just like `send`, except it checks for dropped receivers.
    // Might block if the channel is full.
    pub fn checked_send(msg: T) -> Result<(), NoReceiversError<T>>;
}

struct NoReceiversError<T>(pub T);

How does everyone feel about this? Would that solve the problem?

Alternatively, checked_send could also return an error when the channel is full:

impl<T> Sender<T> {
    // Just like `send`, but never blocks and checks for dropped receivers.
    pub fn checked_send(msg: T) -> Result<(), CheckedSendError<T>>;
}

enum CheckedSendError<T> {
    Full(T),
    NoReceivers(T),
}

I'm personally leaning towards the first suggestion.

Bikesheddable names:

• checked_send
• NoReceiversError
• NoReceivers
• CheckedSendError

[matthieu-m]

It seems to me that checking for dropped receivers is intrinsically complicated, and there are multiple issues to resolve beyond "check on send".

First of all, there's the problem of leaked receivers. Whether they are leaked intentionally (mem::forget), accidentally (Rc/Arc cycle) or functionally (still reachable, but never polled). In either case, merely counting the number of receivers is inadequate to detect that no one is polling the queue. Potential improvements here could be (a) capping the queue or (b) introducing an activity timestamp (whether logical or actual time).

Putting aside leaked receivers, there is still no acknowledgement of processing (by default). A sender can check that a receiver exists when it enqueues an item, but cannot guarantee that a receiver will eventually process such item: there is a data-race between checking whether the receiver is still alive and the receiver's death, and it could very well die after sending was successful.

Which brings us to a third problem; even if a sender detects, upon sending, that no receiver exists any longer, all it can do is avoid enqueuing more items. What of all the items that were already queued? Should there be an API to get them back? Isn't that akin to creating a new receiver?

And of course, there is the issue mentioned that send in a select! is not amenable to returning a Result.

---

As usual in this case, I'd say we should listen to our MOMs (1).

The concern here is reminiscent of guaranteed delivery. In order to implement guaranteed delivery, MOMs will usually provide a feature to address this issue of queued items no longer being deliverable: a dead letter queue.

In this case, I am wondering whether a dead letter callback would be suitable. The creator of the queue would be able to register a dead letter callback of the form Fn(Item) -> () + Send + 'static, which will be used to handle items when there is no receiver, or when receivers have been inactive for too long.

The callback would be triggered on the sender thread whenever a send is called and its activation criteria is met, first dequeuing the items enqueued, and then processing the new item that send wished to send.

(1) Message-Oriented Middleware, such as MQ & co.

---

If we look at the 4 problems I gave above, a dead letter callback solves them all:

1. Leaked receivers are detected by the inactivity detection mechanism, which can be used to invalidate/poison receivers before proceeding to empty the queue and divert further enqueuing.
2. Unprocessed items have a last chance to be processed; simple default callbacks could panic or log.
3. Unprocessed items can be returned to a sender and stored in persistent storage (best effort processing).
4. It works with select!, at the cost of being slightly indirect.

Note: the ability of last-chance processing of queued items does not lead to guaranteed delivery, should the process crash in the middle, any in-memory item is lost. Guaranteed delivery can only be implemented with point-to-point acknowledgement between persistent stores up until the final destination (which can either store or process the item before acknowledging).

---

Alternatives:

• Bounded Queues: bounded queues are great to avoid memory exhaustion, and useful to create back-pressure. They are not suited to detecting inactive/dead receivers. The problem is that by design a bounded queues must be sized for peak time, so that during down time it could take seconds to minutes to actually fill up a queue and realize something is wrong on the other end.
• Checked Send: a checked_send method is better at detecting dead receivers, though the user must realize that there just because an item is enqueued does not mean it will be processed. The mechanism can be adapted to detect inactive receivers.
• Others?

[vorner]

My motivation for bringing the issue up was mostly for accidental subtle bugs that happen on rare cases. In other words, protection against not handling situation you don't know you should handle.

In that light, I don't think adding another sending method that is more complex and people will probably not be interested in doesn't solve my worry ‒ people who don't know about the potential problem won't look for the method and won't use it, still getting the subtle bugs.

That being said, the method does look useful for some use cases where you already do know about the possibility of lost receivers.

[BurntSushi]

Yeah, I think my position is mostly unchanged here. I do like @vorner's comment in that opportunistic deadlock detection might be possible, and that seems like a fine idea to me.

Ok, here's another idea. Maybe we could get back to the old interface for send and recv methods, where dropping receivers closes the channel:

impl<T> Sender<T> {
    fn send(&self, msg: T) -> Result<(), SendError<T>>;
    fn try_send(&self, msg: T) -> Result<(), TrySendError<T>>;
}

impl<T> Receiver<T> {
    fn recv(&self, msg: T) -> Result<T, RecvError>;
    fn try_recv(&self, msg: T) -> Result<T, TryRecvError>;
}

Now the question is what happens to select!. I think we could have the following syntax:

let r: Receiver<T> = ...;
let rs = Vec<Receiver<T>> = ...;

let s: Sender<T> = ...;
let ss = Vec<Sender<T>> = ...;

select! {
    res = recv(r) => match res {
        Ok(msg) => println!("received {}", msg),
        Err(_) => println!("channel closed"),
    }

    (res, r) = recv(rs) => match res {
        Ok(msg) => println!("received {} from {:?}", msg, r),
        Err(_) => println!("channel {:?} is closed", r),
    }

    res = send(s, foo) => match res {
        Ok(()) => println!("sent message"),
        Err(msg) => println!("couldn't send {} because the channel is closed", msg),
    }

    (res, s) = send(ss, foo) => match res {
        Ok(()) => println!("sent message into {:?}", s),
        Err(msg) => println!("couldn't send {} into {:?} because it is closed", msg, s),
    }
}

But you don't have to use match in every case. The following would be supported, too:

select! {
    msg = recv(r) => println!("recv result: {:?}", msg),
    res = send(s, "message") => res.unwrap(),
}

This is not too verbose and seems pretty intuitive. What do you think?

[vorner]

Yes, I think this looks mostly good ‒ at least to me :-).

There are probably some questions to answer, though. Let's say there are two channels I want to send into, one of them lost all of its receivers, the other one is ready to accept. Are they both considered ready?

Let's say there are two channels I want to send into, one of them lost all of its receivers, the other one is ready to accept. Are they both considered ready?

Yes, they are both ready and a random one of those two operations will be completed.

Think about it this way: send(s, msg) is equivalent to s.send(msg), and select! will simply wait until any one of the declared function calls returns. (Of course, only one function call is actually executed.)

If you'd like to skip operations that send into closed channels, you can achieve that as follows:

let mut s1 = Some(s1);
let mut s2 = Some(s2);
// Note that `Option<Sender<T>>` implements `IntoIterator<Item = Sender<T>>`
loop {
    select! {
        (res, _) = send(s1, msg) => match res {
            Ok(()) => break,
            Err(_) => s1 = None,
        }
        (res, _) = send(s2, msg) => match res {
            Ok(()) => break,
            Err(_) => s2 = None,
        }
    }
}

This is a common idiom in Go.