Skip to content

Conversation

@petyaslavova
Copy link
Collaborator

Fix RuntimeError in ClusterPubSub sharded message generator

Problem: RuntimeError: dictionary changed size during iteration when subscribing to multiple sharded channels on different cluster nodes.

Solution: Create a snapshot of node_pubsub_mapping.values() before iteration in _pubsubs_generator() to prevent errors from concurrent modifications.

Changes:
Fixed _pubsubs_generator() in ClusterPubSub class
Added 3 cluster-only tests validating multi-node sharded pubsub and the concurrent mapping changes fix

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a RuntimeError: dictionary changed size during iteration that occurs when subscribing to multiple sharded channels across different cluster nodes. The issue arose because the generator was iterating directly over node_pubsub_mapping.values() while concurrent subscriptions could modify the dictionary.

Key changes:

  • Modified _pubsubs_generator() to create a snapshot of pubsub instances before iteration
  • Introduced cycle boundaries using None markers to properly track iteration completion
  • Added comprehensive tests validating multi-node sharded pubsub behavior

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
redis/cluster.py Fixed generator to snapshot mapping values before iteration and added cycle markers
tests/test_pubsub.py Added three cluster-specific tests for multi-node sharded pubsub scenarios

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@petyaslavova petyaslavova force-pushed the ps_fix_runtime_error_for_sharded_pubsub branch from d76f170 to 50f8ab3 Compare December 17, 2025 09:35
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@petyaslavova petyaslavova merged commit 12c9a38 into master Dec 17, 2025
72 of 73 checks passed
@petyaslavova petyaslavova deleted the ps_fix_runtime_error_for_sharded_pubsub branch December 17, 2025 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants