Skip to content

Run multiple event persisters when using Redis #972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 13, 2020
+123 −34
Merged

Run multiple event persisters when using Redis #972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
51d24eb
Run multiple event persisters when using Redis
erikjohnston Oct 8, 2020
935782b
Fix flakey test
erikjohnston Oct 9, 2020
b18ed7c
Merge remote-tracking branch 'origin/develop' into erikj/periodic_poke
erikjohnston Oct 9, 2020
2fab331
Add comment
erikjohnston Oct 13, 2020
8d4d21d
Damn typos
erikjohnston Oct 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 83 additions & 3 deletions lib/SyTest/Homeserver/Synapse.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,14 @@ sub _init
frontend_proxy_metrics => main::alloc_port( "frontend_proxy[$idx].metrics" ),
frontend_proxy_manhole => main::alloc_port( "frontend_proxy[$idx].manhole" ),

event_persister1 => main::alloc_port( "event_persister1[$idx]" ),
event_persister1_metrics => main::alloc_port( "event_persister1[$idx].metrics" ),
event_persister1_manhole => main::alloc_port( "event_persister1[$idx].manhole" ),

event_persister2 => main::alloc_port( "event_persister2[$idx]" ),
event_persister2_metrics => main::alloc_port( "event_persister2[$idx].metrics" ),
event_persister2_manhole => main::alloc_port( "event_persister2[$idx].manhole" ),

haproxy => main::alloc_port( "haproxy[$idx]" ),
};
}
Expand Down Expand Up @@ -290,14 +298,18 @@ sub start
) : (),

instance_map => {
"frontend_proxy1" => {
"event_persister1" => {
host => "$bind_host",
port => $self->{ports}{event_persister1},
},
"event_persister2" => {
host => "$bind_host",
port => $self->{ports}{frontend_proxy},
port => $self->{ports}{event_persister2},
},
},

stream_writers => {
events => $self->{redis_host} ne '' ? "frontend_proxy1" : "master",
events => $self->{redis_host} ne '' ? [ "event_persister1", "event_persister2" ] : "master",
},

# We use a high limit so the limit is never reached, but enabling the
Expand Down Expand Up @@ -970,6 +982,74 @@ sub _start_synapse
push @worker_configs, $background_worker_config;
}

{
my $event_persister1_config = {
"worker_app" => "synapse.app.generic_worker",
"worker_name" => "event_persister1",
"worker_pid_file" => "$hsdir/event_persister1.pid",
"worker_log_config" => $self->configure_logger("event_persister1"),
"worker_replication_host" => "$bind_host",
"worker_replication_port" => $self->{ports}{synapse_replication_tcp},
"worker_replication_http_port" => $self->{ports}{synapse_unsecure},
"worker_main_http_uri" => "http://$bind_host:$self->{ports}{synapse_unsecure}",
"worker_listeners" => [
{
type => "http",
resources => [{ names => ["client", "replication"] }],
port => $self->{ports}{event_persister1},
bind_address => $bind_host,
},
{
type => "manhole",
port => $self->{ports}{event_persister1_manhole},
bind_address => $bind_host,
},
{
type => "http",
resources => [{ names => ["metrics"] }],
port => $self->{ports}{event_persister1_metrics},
bind_address => $bind_host,
},
],
};

push @worker_configs, $event_persister1_config;
}

{
my $event_persister2_config = {
"worker_app" => "synapse.app.generic_worker",
"worker_name" => "event_persister2",
"worker_pid_file" => "$hsdir/event_persister2.pid",
"worker_log_config" => $self->configure_logger("event_persister2"),
"worker_replication_host" => "$bind_host",
"worker_replication_port" => $self->{ports}{synapse_replication_tcp},
"worker_replication_http_port" => $self->{ports}{synapse_unsecure},
"worker_main_http_uri" => "http://$bind_host:$self->{ports}{synapse_unsecure}",
"worker_listeners" => [
{
type => "http",
resources => [{ names => ["client", "replication"] }],
port => $self->{ports}{event_persister2},
bind_address => $bind_host,
},
{
type => "manhole",
port => $self->{ports}{event_persister2_manhole},
bind_address => $bind_host,
},
{
type => "http",
resources => [{ names => ["metrics"] }],
port => $self->{ports}{event_persister2_metrics},
bind_address => $bind_host,
},
],
};

push @worker_configs, $event_persister2_config;
}

my @base_synapse_command = $self->_generate_base_synapse_command();
my $idx = $self->{hs_index};

Expand Down
71 changes: 40 additions & 31 deletions tests/30rooms/60version_upgrade.pl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1070,46 +1070,55 @@ sub upgrade_room_synced {
$remote_joiner, $new_room_id, ( server_name => $creator->server_name, )
);
})->then(sub {
do_request_json_for( $creator,
method => "GET",
uri => "/r0/publicRooms",
);
})->then( sub {
# Check public rooms list for local user
my ( $body ) = @_;
# The room list can be updated asynchronously, so we retry if it
# doesn't match what we expect.
retry_until_success {
Copy link
Member

@richvdh richvdh Oct 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these changes necessary? And if they are really necessary, can we limit the number of times we retry, rather than going on forever?

Copy link
Member Author

@erikjohnston erikjohnston Oct 13, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test was flakey. This may no longer be necessary, but in general I don't think the public room lists get updated immediately (as they can be populated via background processes).

The retry_until_success will get killed by the test timeout, which is how we use retry_until_success/repeat_until_true elsewhere?

Copy link
Member

@richvdh richvdh Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add a comment to say that we're retrying because the directory is updated asynchonously, or sth.

My main gripe with retry_until_success is that it turns test failures into test timeouts, which we tend to ignore as "oh we had a slow VM for random reasons", so a limit on the number of retries is preferable if possible.

The other problem with retry_until_success is that it swallows the failure, so you can't see what's going wrong. https://github.com/matrix-org/sytest/blob/develop/tests/30rooms/04messages.pl#L387-L408 includes a bunch of code to log what's going on, but on closer inspection maybe we should put all that boilerplate into retry_until_success (which could maybe also benefit from a "try N times" param which defaults to something other than "infinity").

Copy link
Member Author

@erikjohnston erikjohnston Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have added a comment.

Yeah, I think that sounds like a general issue with retry_until_success. We use this pattern a lot so it feels like something to investigate and improve separately.

do_request_json_for( $creator,
method => "GET",
uri => "/r0/publicRooms",
)->then( sub {
# Check public rooms list for local user
my ( $body ) = @_;

log_if_fail "Public rooms list for local user", $body;
log_if_fail "Public rooms list for local user", $body;

assert_json_keys( $body, qw( chunk ) );
assert_json_keys( $body, qw( chunk ) );

# Check that the room list contains new room id
any { $new_room_id eq $_->{room_id} } @{ $body->{chunk} }
or die "Local room list did not include expected room id $new_room_id";
# Check that the room list contains new room id
any { $new_room_id eq $_->{room_id} } @{ $body->{chunk} }
or die "Local room list did not include expected room id $new_room_id";

# Check that the room list doesn't contain old room id
none { $room_id eq $_->{room_id} } @{ $body->{chunk} }
or die "Local room list included unexpected room id $room_id";
# Check that the room list doesn't contain old room id
none { $room_id eq $_->{room_id} } @{ $body->{chunk} }
or die "Local room list included unexpected room id $room_id";

do_request_json_for( $remote_joiner,
method => "GET",
uri => "/r0/publicRooms",
);
})->then( sub {
# Check public rooms list for remote user
my ( $body ) = @_;
})
}
})->then(sub {
# The room list can be updated asynchronously, so we retry if it
# doesn't match what we expect.
retry_until_success {
do_request_json_for( $remote_joiner,
method => "GET",
uri => "/r0/publicRooms",
)->then( sub {
# Check public rooms list for remote user
my ( $body ) = @_;

log_if_fail "Public rooms list for remote user", $body;
log_if_fail "Public rooms list for remote user", $body;

assert_json_keys( $body, qw( chunk ) );
assert_json_keys( $body, qw( chunk ) );

# Check that the room list contains new room id
any { $new_room_id eq $_->{room_id} } @{ $body->{chunk} }
or die "Remote room list did not include expected room id $new_room_id";
# Check that the room list contains new room id
any { $new_room_id eq $_->{room_id} } @{ $body->{chunk} }
or die "Remote room list did not include expected room id $new_room_id";

# Check that the room list doesn't contain old room id
none { $room_id eq $_->{room_id} } @{ $body->{chunk} }
or die "Remote room list included unexpected room id $room_id";
# Check that the room list doesn't contain old room id
none { $room_id eq $_->{room_id} } @{ $body->{chunk} }
or die "Remote room list included unexpected room id $room_id";

Future->done( 1 );
Future->done( 1 );
})
}
});
}