Redis queue - concurrency issues

[girtsu]

I noticed that moveExpired method can cause problems when multiple workers are running simultaneously. If multiple workers call zrevrangebyscore at the same time, before zremrangebyscore is called, both will get the same message IDs and will add them to the waiting queue, causing duplicate entries.

If this happens, one of two problems will occur:
a) Duplicate message ID will be fetched after the job is already processed, causing queue to crash with this exception:
PHP Notice 'yii\base\ErrorException' with message 'Undefined offset: 1' in vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php:106
This happens because the payload was already deleted and the script doesn't check if it actually got the payload before trying to parse it.
b) Two or more worker processes will get the same duplicated message ID, causing the same job to be executed multiple times in parallel.

Steps to repeat:

1. Launch multiple workers (4+ recommended) that listen to the queue with short wait time
2. Add a couple thousand jobs to the queue with a delay(30s)
3. Wait for errors

[girtsu]

Note - I noticed this when using version 2.0.0
Looks that this was already addressed in 2.0.1 by this commit 8c9555c

However wouldn't it be safer to limit the amount of items to be processed at once? Otherwise we could run into out of memory errors if a million jobs are scheduled to run at the same time.
Something like this:

$chunkSize = 1000;
do {
    $expired = $this->redis->zrevrangebyscore($from, $time, '-inf', 'LIMIT', 0, $chunkSize);
    foreach ($expired as $id) {
        //Add only on successful removal
        if ($this->redis->zrem($from, $id)) {
            $this->redis->rpush("$this->channel.waiting", $id);
        }
    }
} while ($expired);

[zhuravljov]

Yes, the bug was caught through #118 and fixed for 2.0.1.

yii2-queue/src/drivers/redis/Queue.php

Lines 143 to 146 in 5e3bc6c

	if ($this->redis->set("$this->channel.moving_lock", true, 'NX', 'EX', 1)) {
	$this->moveExpired("$this->channel.delayed");
	$this->moveExpired("$this->channel.reserved");
	}

This code locks moving for 1 second, and only one of all workers will move reserved and delayed messages. I think, that time is enough to move expired messages into waiting list.

[aziza-kamet]

I still have this issue, when launching 8 workers, with repeat set to false. Also ttr = 5, attempts = 5.

PHP Notice 'yii\base\ErrorException' with message 'Undefined offset: 1'
in /vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php:155

Could someone help, please? I think I'm doing something wrong.

[zhuravljov]

What version of the queue are you using? Redis version? Also add full stack trace of the error please.

[aziza-kamet]

yiisoft/yii2-queue: 2.0.2
yiisoft/yii2-redis: 2.0.8
redis: 4.0.9

'yii\base\ErrorException' with message 'Undefined offset: 1'

in /Users/rocketman27/dev/banners/vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php:155

Stack trace:
#0 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php(155): yii\base\ErrorHandler->handleError(8, 'Undefined offse...', '/Users/rocketma...', 155, Array)
#1 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php(61): yii\queue\redis\Queue->reserve(0)
#2 [internal function]: yii\queue\redis\Queue->yii\queue\redis\{closure}(Object(yii\queue\cli\SignalLoop))
#3 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2-queue/src/cli/Queue.php(108): call_user_func(Object(Closure), Object(yii\queue\cli\SignalLoop))
#4 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php(71): yii\queue\cli\Queue->runWorker(Object(Closure))
#5 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2-queue/src/drivers/redis/Command.php(56): yii\queue\redis\Queue->run(false)
#6 [internal function]: yii\queue\redis\Command->actionRun()
#7 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#8 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#9 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2/console/Controller.php(148): yii\base\Controller->runAction('run', Array)
#10 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('run', Array)
#11 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('queue/run', Array)
#12 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2/console/Application.php(147): yii\console\Application->runAction('queue/run', Array)
#13 /Users/rocketman27/dev/banners/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(yii\console\Request))
#14 /Users/rocketman27/dev/banners/yii(23): yii\base\Application->run()
#15 {main}

[askobara]

same here:

2018-06-12 14:01:03 [pid: 22488] - Worker is stopped (0:12:18)
PHP Notice 'yii\base\ErrorException' with message 'Undefined offset: 1'

in /etc/bamboo/bamboo.home/xml-data/build-dir/FX-FXAPID43-JOB1/releases/140/vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php:152