consumer stops consuming events and throws KinesisMessageDrivenChannelAdapter : The lock for key 'xxxxxxxx:shardId-00000000000X' was not renewed in time

[malvine]

Hi,

I've got interesting use case and it's very simple to replicate.
I am using the latest binder version.

My setup:

• One Kinesis stream with two shards.
• One consumer app with 3 instances let's call them instance1, instance2, instance3.
• Latest binder version.

To replicate:

1. Start the consumer: usually one instance gets all both shards as owner. Let's say instance1 gets all the shards. So far so good. Instance consuming events.

2. Restart the active consuming instance e.g. instance1. Instance2 will get now shard-0 and instance3 gets shard-1 as owner. (This step might take couple of attempts as sometimes instance2 or instance3 gets all the shards. Keep trying until you get one shards per instance)

3. Once you got instance1 - doing nothing; instance2 owner of shard-0 and instance-3 owner of shard-1, that's when the instances stop consuming any events and both of them have this in the logs every 30 seconds:

.i.k.KinesisMessageDrivenChannelAdapter : The lock for key 'xxxxxxxx:shardId-000000000002' was not renewed in time java.util.concurrent.TimeoutException: null 	at 
java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) ~[na:na]
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.renewLockIfAny(Kin
esisMessageDrivenChannelAdapter.java:1030) ~[spring-integration-aws-2.4.0.jar:na] 
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.execute(KinesisMes
sageDrivenChannelAdapter.java:946) ~[spring-integration-aws-2.4.0.jar:na] 
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ConsumerDispatcher.run(KinesisMessageDrivenChannelAdapter.java:856) ~[spring-integration-aws-2.4.0.jar:na 

The only way to stop this is to restart one of the consuming instances so all the shards are back to one instance.

The binder version 2.0.1.RELEASE doesn't have this issue.
Pretty sure it's not even the binder issue - it's probably integrations-aws lib issue.

Thanks.

[artembilan]

I think there must be some other logs related to the lock processing.
Something like:

logger.error(ex, () -> "Error during locking: " + lock);

Would you mind to share that one with us as well?

Also: is there a chance to use really the latest 2.2.0 Kinesis binder version: https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-stream-binder-kinesis ?

The spring-integration-aws-2.4.0 is not the latest one...

[malvine]

yeah, sorry was trying with 2.1.0 thought if that one doesn't have the issue.
ok. with the latest version 2.2.0 get exactly the same.

No error before that or after. just that endlessly on both instances.

2022-05-11 17:39:28.046  INFO 99193 --- [esis-consumer-1] a.i.k.KinesisMessageDrivenChannelAdapter : The [ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=AFTER_SEQUENCE_NUMBER, sequenceNumber='49629266667711709536924126896771570517185546512247029778', timestamp=null, stream='xxxxxx', shard='shardId-000000000001', reset=false}, state=NEW}] has been started.
2022-05-11 17:39:39.020  INFO 99193 --- [is-dispatcher-1] a.i.k.KinesisMessageDrivenChannelAdapter : The lock for key 'xx:xxxxxx:shardId-000000000001' was not renewed in time

java.util.concurrent.TimeoutException: null
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1950) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2085) ~[na:na]
	at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.renewLockIfAny(KinesisMessageDrivenChannelAdapter.java:1031) ~[spring-integration-aws-2.5.1.jar:na]
	at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumer.execute(KinesisMessageDrivenChannelAdapter.java:947) ~[spring-integration-aws-2.5.1.jar:na]
	at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ConsumerDispatcher.run(KinesisMessageDrivenChannelAdapter.java:857) ~[spring-integration-aws-2.5.1.jar:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]

[artembilan]

OK. Any chances that you can share a simple project to reproduce?
Is it possible for me to run it against LocalStack then?

[malvine]

Yeah, I did put a quick project together and seems like on LocalStack you can easy to replicate the issue. Once you get shards being consumed by not one instance but two.
Here is the link

[artembilan]

Hi @malvine !

Thank you for the sample.
I definitely could reproduce it locally with my Docker.
It's impossible to debug it, but at least I see where we are heading.
So, I guess we can explain the problem like this:
As long as consumer doesn't take all the shards ownership from the stream, it doesn't consume even from a shard it owns, neither update the lock for the shard in DynamoDb.
I mean it looks like lock manager is broken for this partial situation somehow...

Will look into code today.

[artembilan]

Well, I'm not sure, but it looks like the problem is here:

    /**
     * Setting this flag to true will prevent the thread from being blocked (put to sleep) for the lease duration and
     * instead will return the call with the lock not granted exception back to the caller. It is up to the caller to
     * optionally back-off and retry and to acquire the lock.
     */
    private final boolean shouldSkipBlockingWait;

Which we don't set in our DynamoDbLockRegistry and then in the KinesisMessageDrivenChannelAdapter.ShardConsumerManager we do block on this line:

if (lock.tryLock()) {
	this.locks.put(key, lock);
}

So, if another instance is holder for the lock, we don't return immediately as it states for the tryLock() contract, but rather step into a busy-wait loop in the AmazonDynamoDBLockClient.acquireLock().

Does it make sense?

I'll try to fix in my local Spring Integration AWS copy and re-run your solution to verify.

[artembilan]

So, yeah... My assumption and investigation is correct.
The tryLock() contraction must be honored in the DynamoDbLockRegistry.
Otherwise the AmazonDynamoDBLockClient steps into a busy-wait loop and never fulfill a Future for the lock.

I'm going to push the fix today, but if you need a release for Spring Integration AWS 2.5.2, then I can do that tomorrow.

I'm moving this to Spring Integration AWS project though...

[malvine]

@artembilan thank you very much!

[postalservice14]

Did this get resolved? I'm running into this exact issue - even when upgrading Spring Integration AWS to 2.5.4.

I was even still able to reproduce it with the quick project that @malvine made using Spring Integration AWS to 2.5.4.

[artembilan]

@postalservice14 ,

can you share, please, an updated sample project to see what is going on?
I mean how are you sure that you have updated Spring Integration AWS to 2.5.4?
There is no explicit dependency for that in the mentioned project.
Not a fact that it is not overridden transitively by Kinesis Binder provided.

[postalservice14]

@artembilan thanks for the quick reply! Here's a fork of his project upgraded to 2.5.4 (you can see the latest commit). And I can still get it to happen using his instructions in the readme.

https://github.com/postalservice14/kinesis-lock-was-not-renewed-in-time

[artembilan]

@postalservice14 ,

any chances that you can upgrade your solution (not that sample app) to the latest Spring Boot and org.springframework.cloud:spring-cloud-stream-binder-kinesis:3.0.0 ?
There is no @EnableBinding and friends any more in Spring Cloud Stream: everything has to be mapped via java.util.function primitives.

I'll try to reproduce over weekend in a simple unit test with Testcontainers.
I also don't see a reason in a producer side for that sample: according to the issue description we got a problem just with shards assignments - no need in any data!

Well, I'd prefer to have a new GH issue with fresh, actual and relevant info instead of trying to resurrect this issue against out-dated versions.
Thanks for understanding!