Reconnect does not work with large data sets

[vsespb]

use strict;
use warnings;
use Redis;
my $redis = Redis->new(
server => 'localhost:6379',
reconnect=>10,
every => 100_000,
cnx_timeout   => 10,
read_timeout  => 3,
write_timeout => 3,
#on_connect => sub { while ($_[0]->info("persistence")->{loading}) {sleep 1 } }
);

for (1..100_000) { # adjust me
  $redis->set("test_$_", "x" x 10_000);
}

print STDERR "Please restart redis\n";

while () {
  $redis->get("test_1");
  sleep 1;
}

crashes with
Error while reading from Redis server: at /usr/local/share/perl/5.14.2/Redis.pm line 267.
or
[get] LOADING Redis is loading the dataset in memory, at /usr/local/share/perl/5.14.2/Redis.pm line 267.

does not work even if I uncomment on_conenect callback. possible intermittent issue.

related: shogo82148/Redis-Fast#19

[billmoseley]

I'm using Resque to push jobs and Sentinel for HA and also seeing this same error at line 267 when a master is shut down (before the failover starts).

I'm not submitting large data, rather I see it when I'm pushing a lot of small jobs very quickly. Adding a sleep 0.1 between push calls seems enough to make it stop. I suspect it's not the sleep that is making the difference but that I'm making frequent calls.

When the error happens both my client and worker croak at about the same time.

Should this really croak in this case or should __throw_reconnect() be called instead?

redis-server 2.8.13 on OS X and Redis 1.975

[vsespb]

UPD:

workaround
on_connect => sub { while ($_[0]->info("persistence")->{loading}) {sleep 1 } }
could be better written as this

on_connect => sub {
  my ($redis) = @_;
  local $@;
  while ( !eval{ $redis->exists("some-non-existing-key");1 } && $@ =~ /LOADING/ ) { usleep 100_000; }
}

because INFO is pretty slow. trick with EXISTS is not well documented, but seems Ok:
https://groups.google.com/d/msg/redis-db/vZhfTkjus2I/ZYgkivs75AYJ

[melo]

Its a pity that Redis don't have a command that blocks until the server is ready…

But I think having a official flag to "wait until is loaded" that you can use in the call to new() would make client code saner. Even if inside we do the last "on_connect" suggestion from @vsespb. At least, if a better way comes along, we already have the API in place.