Sync stream is either missing echo of events or is sending them earlier than anticipated

[turt2live]

On March 31st (roughly) there was an uptick of "message stuck as latest message" reports to the riot clients (riot-web's upstream issue is element-hq/element-web#10032 for this and other cases).

The cause looks to be because Synapse is not sending events down /sync sometimes for events we've sent. There are also some reports of events just not being received on the other end which could be related. Most reporters were on matrix.org, and the reports seemed to stop once matrix.org rolled back.

Reproduction steps appear to be time-dependent, and possibly only useful for matrix.org-sized homeservers:

1. Send an event
2. Start a /sync request
3. Send another event
4. Receive event ID for first event (/send/m.room.message completes)
5. Receive event down /sync
6. Send yet another event
7. Start a /sync request
8. Receive event ID for second and third events
9. Receive the third event down /sync, but not the second (sometimes the other way around)
10. Never receive the second event through /sync, making it stuck on Riot

Most easily reproduced during high traffic volumes on matrix.org, though one case was also reproduced at ~22:00 UTC yesterday (but never again that night).

Theory at least from my side is that the request hits a busy synchrotron which advances the stream token past the second event to the third event, invoking amnesia.

@erikjohnston's conclusions appear to be that synapse might be sending the event down /sync ahead of the /send/m.room.message request completing?

[turt2live]

This rageshake is my ~22:00 UTC attempt: https://github.com/matrix-org/riot-web-rageshakes/issues/2501

(I think, I might have gotten the timestamp wrong).

[richvdh]

@erikjohnston's conclusions appear to be that synapse might be sending the event down /sync ahead of the /send/m.room.message request completing?

this is valid behaviour though.

[turt2live]

this is valid behaviour though.

Yup, and it seems like Riot is okay with this. My attempt revealed no syncs with the event ID contained within, aside from a sync which had the second user's read receipt for the event. The event ID did not appear early or late.

[richvdh]

well there are a boatload of errors in the logs which suggest that some replication updates aren't being processed correctly. They include things like:

Traceback (most recent call last):
  File "/home/synapse/src/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 254, in handle_command
    await handler(cmd)
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 650, in on_POSITION
    [stream.parse_row(update[1]) for update in updates],
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 650, in <listcomp>
    [stream.parse_row(update[1]) for update in updates],
IndexError: string index out of range

AttributeError: 'GenericWorkerSlavedStore' object has no attribute 'get_all_push_rule_updates'

Traceback (most recent call last):
  File "/home/synapse/src/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 254, in handle_command
    await handler(cmd)
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 668, in on_POSITION
    await self.handler.on_rdata(cmd.stream_name, rows[-1].token, rows)
AttributeError: 'DeviceListsStreamRow' object has no attribute 'token'

(also PushersStreamRow, EventsStreamRow, ToDeviceStreamRow, probably others.)

My inclination would be to fix those, then redeploy and see if the problem is magically cured.

[turt2live]

Logging for riot-web added here: matrix-org/matrix-js-sdk#1300

Might help narrow it down, maybe.

[richvdh]

For the record, we rolled back matrix.org's synapse deployment on 2nd April while we investigated these problems.

As of 11:06 UTC this morning, we have redeployed matrix.org with current develop. We've fixed the obvious exceptions being spewed by the replication connections, so I am optimistic that this problem has gone away; however, we never really got to the bottom of what was going on, so it's entirely possible there are still problems.

In other words: feedback welcome as to whether this bug seems to be fixed or not.