Stuck notifications

[Johennes]

We are experiencing multiple issues in the area of "stuck" notifications and unread markers. Many are related to the way receipts interact with threaded messages.

Symptoms

We are seeing different symptoms of the problem:

• "Stuck" unread dots and notification counters: Even when I have read all messages in a room, the room is still marked as unread or having notifications.
• Unread dots or notification counters returning on app startup: When I restart the app (or refresh the page on Web) rooms which I have already read re-appear as unread.
• Notification counters growing and shrinking spontaneously after entering or scrolling in a room.

Spec-level causes

Message ordering

Fundamentally, in order to interpret the meaning of a receipt that says "I have read everything up to here", we need to know what order messages are in. This is not clear in the spec, and we propose to make it clear and explicit in MSC4033.

In the meantime, Element Web uses a combination of "sync" order (the implicit order of events arriving via a /sync request) and "timestamp" order (using the ts property within events).

Some of the existing bugs are probably caused by this inconsistency, but it is not clear yet how many: we believe there are also bugs in the implementation that cause additional problems, and this theoretical inconsistency is only the cause of a few problems.

Which thread the root belongs to

The spec has what we consider a bug when it talks about which thread the root message belongs to, which has been reflected in client code, making it inconsistent with the server implementation (at least on the Synapse server). We have a proposal to fix this bug in MSC4037.

Identifying which thread any message is in

It is sometimes difficult for clients to identify which thread an event belongs to, meaning that a receipt pointing to it is sometimes ignored. We have begun drafting MSC4023 to address this.

Other

Previously, we believed that MSC3981 (recursive relations) would solve some of the problems, but since that MSC does not solve the event-ordering problem (because the events from the /relations API are returned in "topological" order) we no longer believe it is important, except as a performance optimisation.
Code-level causes

Code-level causes

We have found and fixed several bugs in the Element Web code that were caused by an incomplete understanding of the meaning of threaded and unthreaded read receipts. We anticipate that some more exist.

(We believe that the primary reason why we're not seeing the same problems on mobile is that the apps persist events they've received whereas Element Web has to re-fetch from scratch after every launch. As a result, any issue in the unread state logic, strikes again and again. The apps also use a single timeline whereas Element Web maintains one timeline per thread in addition to the main timeline in every room.)

High-level plan of actions

### Tasks
- [x] Set up POC branch / deployment with threads disabled – https://github.com/vector-im/element-web/issues/25676
- [x] Fix read-receipt behavior around thread roots – https://github.com/matrix-org/matrix-js-sdk/pull/3600
- [x] Fix read-receipt behavior around non-thread relations to thread roots – https://github.com/matrix-org/matrix-js-sdk/pull/3607
- [x] Fix missing message issues due to replies to unknown events – https://github.com/matrix-org/matrix-js-sdk/pull/3615
- [x] Fix unread count returning from zero after reload – https://github.com/vector-im/element-web/issues/25806
- [x] Fix notification counters sometimes being doubled – https://github.com/vector-im/element-web/issues/25803
- [x] Set up unread & notification test suite – https://github.com/vector-im/element-web/issues/25449
- [ ] Unthreaded receipt is not returned in the next syncs, confirm bug and get fixed - https://github.com/element-hq/synapse/issues/17247
- [ ] Pre-investigate issues listed after the threads model refactoring to see if any of them are fixable before
- [ ] Refactor threads model code and remove serverless / legacy code path to simplify further changes
- [ ] Fix missing notifications – https://github.com/vector-im/element-web/issues/25621
- [ ] Fix incorrect thread reply count – https://github.com/vector-im/element-web/issues/24636
- [ ] Fix notification counters not adding up between spaces and rooms – https://github.com/vector-im/element-web/issues/20372
- [ ] Fix unread counter explosion – https://github.com/vector-im/element-web/issues/25479
- [ ] Fix Zombie notifications from old threads – ? issue ?
- [ ] Fix read receipts / fully read marker jumping backwards – ? issue ?
- [ ] Triage existing issues to increase confidence that the root causes will actually be fixed through the MSCs mentioned above.
- [ ] Fix remaining bugs found in no-threads POC as they likely impact the threaded experience as well – https://github.com/vector-im/element-web/issues/25676

We believe a lot of progress can still be made without spec changes. So we're slightly deprioritising work on the MSCs.

• MSC4037: Thread root is not in the thread matrix-org/matrix-spec-proposals#4037
  We claim it is implemented in EW
  Synapse will need small fixes when this is merged
• MSC4033: Explicit ordering of events for receipts matrix-org/matrix-spec-proposals#4033
  Needs implementation on server and client but we believe it should mostly only cover edge cases
• MSC4023: Thread ID for 2nd order-relation matrix-org/matrix-spec-proposals#4023
  There is disagreement about the way to specify this and possible interference with MSC3051: A scalable relation format matrix-org/matrix-spec-proposals#3051
• MSC3389: Relation redactions matrix-org/matrix-spec-proposals#3389
  There is likely no good workaround other than implementing this MSC

New issue inbox

The following is a holding area for newly reported issues that require review. Once reviewed, issues should either be moved to one of the other task lists below or, if not applicable, removed from this epic.

### Tasks
- [ ] https://github.com/vector-im/element-web/issues/25331
- [ ] https://github.com/vector-im/element-web/issues/25420
- [ ] https://github.com/vector-im/element-web/issues/25481
- [ ] https://github.com/vector-im/element-web/issues/25480
- [ ] https://github.com/vector-im/element-web/issues/25479
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/21717
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/21724
- [ ] https://github.com/vector-im/element-web/issues/25513
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/21737
- [ ] https://github.com/vector-im/element-web/issues/25482
- [ ] https://github.com/vector-im/element-web/issues/25528
- [ ] https://github.com/vector-im/element-web/issues/25541
- [ ] https://github.com/vector-im/element-web/issues/25621
- [ ] https://github.com/vector-im/element-web/issues/24547
- [ ] https://github.com/vector-im/element-web/issues/25623
- [ ] https://github.com/vector-im/element-web/issues/25642
- [ ] https://github.com/vector-im/element-web/issues/25670
- [ ] https://github.com/vector-im/element-web/issues/25676
- [ ] https://github.com/vector-im/element-web/issues/23976
- [ ] https://github.com/vector-im/element-web/issues/25804
- [ ] https://github.com/vector-im/element-web/issues/25806
- [ ] https://github.com/vector-im/element-web/issues/24111
- [ ] https://github.com/vector-im/element-web/issues/25408
- [ ] https://github.com/vector-im/element-web/issues/25907
- [ ] https://github.com/vector-im/element-web/issues/25904
- [ ] https://github.com/vector-im/element-web/issues/25929
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/22226
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/22363
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/22414
- [ ] https://github.com/vector-im/element-web/issues/25975
- [ ] https://github.com/vector-im/element-web/issues/25984
- [ ] https://github.com/vector-im/element-web/issues/25950
- [ ] https://github.com/vector-im/element-web/issues/26063
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/22526
- [ ] https://github.com/matrix-org/matrix-js-sdk/issues/3665
- [ ] https://github.com/element-hq/element-web/issues/26933

Tasks not blocked by spec work

### Tasks
- [x] https://github.com/vector-im/element-web/issues/24388
- [x] https://github.com/vector-im/element-web/issues/24000
- [x] https://github.com/vector-im/element-web/issues/23991
- [x] https://github.com/vector-im/element-web/issues/23685
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/21667
- [ ] https://github.com/vector-im/element-web/issues/24629
- [ ] https://github.com/vector-im/element-web/issues/25207
- [ ] https://github.com/vector-im/element-web/issues/25196
- [ ] https://github.com/vector-im/element-web/issues/25212
- [ ] https://github.com/vector-im/element-web/issues/10954
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/21641
- [ ] https://github.com/vector-im/element-web/issues/25411
- [ ] https://github.com/vector-im/element-web/issues/25450
- [ ] https://github.com/vector-im/element-web/issues/25449
- [ ] https://github.com/vector-im/element-web/issues/25676
- [ ] https://github.com/vector-im/element-web/issues/25596

Tasks that are related to or dependent on spec work

We've written the following MSCs to try and address the root causes in a reliable and performant way:

• MSC4033: Explicit ordering of events for receipts matrix-org/matrix-spec-proposals#4033
  A client-side workaround for this based on timestamp ordering has been implemented. This is imperfect and easily abused though.
• MSC4023: Thread ID for 2nd order-relation matrix-org/matrix-spec-proposals#4023
  A client-side workaround for this based on calling /event to fetch the parent has been implemented. This is functionally correct but has a noticeable performance impact.
• MSC4037: Thread root is not in the thread matrix-org/matrix-spec-proposals#4037
  The client-side code has already been modeled to reflect the behavior proposed in the MSC (which is also what Synapse already does today).
• MSC3981: /relations recursion matrix-org/matrix-spec-proposals#3981
  We expect this to not help with the ordering problems but it will be a performance improvement.

### Tasks
- [ ] https://github.com/vector-im/element-meta/issues/1350
- [ ] https://github.com/matrix-org/synapse/issues/15377
- [ ] https://github.com/vector-im/element-web/issues/25021
- [ ] https://github.com/vector-im/element-web/issues/24312
- [ ] https://github.com/vector-im/element-web/issues/24394
- [ ] https://github.com/vector-im/element-web/issues/24595
- [ ] https://github.com/vector-im/element-web/issues/24442
- [ ] https://github.com/vector-im/element-web/issues/25408
- [ ] https://github.com/matrix-org/element-web-rageshakes/issues/21679
- [ ] https://github.com/vector-im/element-web/issues/25482
- [ ] https://github.com/vector-im/element-meta/issues/1714
- [ ] https://github.com/matrix-org/synapse/issues/15701
- [ ] https://github.com/vector-im/element-web/issues/25395
- [ ] https://github.com/vector-im/element-web/issues/25893

Issues that are related but out of scope

• Several places in the code use timestamp to determine event order matrix-org/matrix-js-sdk#3325

Time sheeting

WEB: Stuck notifications

[Johennes]

Descoping #24373 as this doesn't have the same level of severity as the other issues.

[Johennes]

As mentioned in the issue description, the remaining bugs will require a spec change. We're currently in the process of drafting the MSC but it'll, unfortunately, be more involved than just a simple client-side bug fix.

[jplatte]

I don't think that is true. If you look at the comments in #24595, you will find some from rooms where threads were never used and IME clearing cache often fixes these issues.

[Johennes]

We're focusing on the most prominent problems first which are in the interplay of threads and other relations. There may be further issues beyond that.

[AlBundy33]

I had the same issue in some rooms.
Because I don't participate that much in this rooms I tried to write a test-message and it seems that the stuck notifications are gone.