Redis cluster - Not handling a Redis master node going down

[gaeljw]

Hello folks,

I'm encountering an issue with the following scenario:

• using Redis Cluster (3 masters + 3 slaves)
• Lettuce configured with complete list of servers (6 servers) through a RedisClusterClient
• Using app with requests that need keys on Redis server 02 (for instance), so far so good
• Stopping Redis server 02
• App requests are now waiting (indefinitely it seems) for server 02 to be back up, requests are not sent to server 05 which is now the master for these keys

I was expecting Lettuce to automatically use server 05 and don't bother anymore with server 02. And/or at least raise errors but not "wait indefinitely".

Logs in the app are like this:

# When stopping server
2023-06-20 08:46:39,003 - [INFO] - from io.lettuce.core.protocol.ConnectionWatchdog in lettuce-eventExecutorLoop-1-1 ?.?:? Reconnecting, last destination was 10.76.x.y:6379
2023-06-20 08:46:39,010 - [WARN] - from io.lettuce.core.protocol.ConnectionWatchdog in lettuce-epollEventLoop-4-2 ?.?:? Cannot reconnect to [10.76.102.236:6379]: finishConnect(..) failed: Connection refused: /10.76.x.y:6379

# ...

# When restarting Redis server
2023-06-20 08:48:09,303 - [INFO] - from io.lettuce.core.protocol.ConnectionWatchdog in lettuce-eventExecutorLoop-1-4 ?.?:? Reconnecting, last destination was 10.76.x.y:6379
2023-06-20 08:48:09,323 - [INFO] - from io.lettuce.core.protocol.ReconnectionHandler in lettuce-epollEventLoop-4-1 ?.?:? Reconnected to 10.76.x.y:6379

In the meantime, requests are "waiting" and never sent to to the new Redis master server.

What am I missing here?

I'm kinda new to both Redis and Lettuce. Any help will be appreciated.

For the record, following code is used:

// Done once (at initialization of the app/service basically)
List<RedisURI> uris = ...
RedisClusterClient client = RedisClusterClient.create(uris);
client.getPartitions(); // Get partitions to ensure that client is ready https://github.com/lettuce-io/lettuce-core/issues/928
Future<StatefulRedisClusterConnection<String, String>> redisConnection = client.connectAsync(StringCodec.UTF8);

// Then, for each request of the app
// (we actually do this with Scala but the idea translates to the following in Java)
Future<String> futureResult = redisConnection.thenCompose(connection -> connection.async().get("..."));

[gaeljw]

Just tried adding a timeout (I though there was one by default actually) with ClusterClientOptions.builder().timeoutOptions(TimeoutOptions.enabled(Duration.ofSeconds(5))).build().

It's better as now the requests do fail after 5s instead of waiting infinitely but still Lettuce is not connecting to the new Redis master.

[gaeljw]

Adding a topology refresh seems to do the trick ClusterTopologyRefreshOptions.builder().enableAllAdaptiveRefreshTriggers() in addition to the timeout.

[gaeljw]

Note that it still takes quite a long time (IMHO) to adapt. Even after using refreshTriggersReconnectAttempts(1), I observe delays ~30s to 60s before adapting.

[mp911de]

App requests are now waiting (indefinitely it seems)

Commands time out after 60 seconds by default. You can control the command timeout via RedisURI. It would be good to capture some debug (or trace) logs of the quiet period to see what is going on during that time vs. what you expect to be happening.