CASSANALYTICS-175: Exclude IP address from RingInstance equality so node replacement does not fail bulk write jobs#216
Open
rustyrazorblade wants to merge 1 commit into
Conversation
…ode replacement does not fail bulk write jobs A node that is replaced and returns with a different IP address (e.g. a pod replacement in Kubernetes) is the same logical instance, but RingInstance equality included the IP address, so CassandraTopologyMonitor reported a topology change and failed S3 bulk write jobs. The same comparison runs in RecordWriter task validation and per-instance consistency accounting. Instance identity is now clusterId, token, fqdn, rack, port and datacenter. Patch by Jon Haddad for CASSANALYTICS-175
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During S3 bulk writes,
CassandraTopologyMonitorpolls the topology every 5 seconds and cancels the job if it differs from the topology captured at job start. The comparison bottoms out inRingInstance.equals, which included the node's IP address.A node that goes down and rejoins with a different IP — routine in Kubernetes, where a rescheduled pod keeps its hostname, host ID, and data but gets a new IP — is the same logical instance with the same token ownership. The write remains correct, but the monitor reported "Topology changed during bulk write" and failed the job. On clusters with hundreds of nodes across several DCs, this makes long-running jobs fail routinely. The same equality is also used by
RecordWriter.validateTaskTokenRangeMappingson every executor task (both transports) and by the per-instance consistency accounting inReplicaAwareFailureHandlerandImportCompletionCoordinator, which counted the old-IP and new-IP instance as two distinct replicas.This patch removes the IP address from
RingInstance.equals/hashCode. Instance identity is now clusterId, token, fqdn, rack, port, and datacenter — sufficient to distinguish nodes, since two live nodes cannot share fqdn + port + datacenter. Sidecar resolves fqdn via reverse DNS and falls back to the IP string when resolution fails, so deployments without DNS see no behavior change. Real topology changes (nodes added, removed, joining, leaving) are still detected through instance membership and pending-state comparison.The IP address was originally added to the equality in CASSANDRA-18852, the same change that introduced building
RingInstancefromReplicaMetadata, which carries no token — leaving the IP as a stand-in discriminator.JIRA: https://issues.apache.org/jira/browse/CASSANALYTICS-175