12.10.2026 Problem:
Clients reported write queries failing with timeout errors against an Apache Cassandra DC despite all production nodes appearing UP. Application and database logs showed request failures with “Unexpected exception during request” and quorum-level write timeouts. System logs contained FailureDetector warnings about not marking nodes down due to large local pauses and MessagingService warnings about incompatible gossip versions. Nodetool outputs showed a mix of nodes in the ring (single DC with five active nodes), a release version from the 2.2.x series, and tpstats reporting dropped RANGE_SLICE messages.
Configuration context: the nodes were running OpenJDK 1.8 with cassandra-env configured for an extremely large heap (MAX_HEAP_SIZE=”100G”, HEAP_NEWSIZE=”25G”) and the JVM process on production hosts showed -Xms100G -Xmx100G -Xmn25G alongside CMS tuning flags (CMSInitiatingOccupancyFraction, MaxTenuringThreshold=1, CMSWaitDuration=10000 and detailed GC logging enabled).
Process:
Step 1: Artifact collection and initial observation
Client uploaded system.log, gc logs, cassandra-env and nodetool outputs. The uploaded artifacts were inspected to reconcile the reported write timeouts with observed node health. The logs confirmed recurring request failures, FailureDetector messages about long local pauses, and MessagingService warnings referencing gossip-version mismatches; these symptoms supported a hypothesis of node-side stalls rather than network partitioning.
Step 2: GC log analysis
GC logs were examined for pause times and generation activity. Several ParNew/CMS cycles showed stop-the-world pauses measured in the hundreds to thousands of milliseconds with steady growth of Old Gen between collections. From object-size deltas and the configured Xmx/Xms values in the JVM command line, the heap was inferred at ~100 GB; these long pauses aligned temporally with the FailureDetector warnings and request errors and explained transient unresponsiveness that would cause quorum write failures.
Step 3: JVM and cassandra-env configuration review
Process listings and cassandra-env were reviewed next. The production JVM used a 100 GB heap with a 25 GB NewGen and had MaxTenuringThreshold set to 1 and an explicit CMSWaitDuration. This configuration promotes premature promotion of short-lived objects to Old Gen and makes CMS work harder on a very large heap, increasing the probability of long stop-the-world pauses. Hard-sizing Xms/Xmx to 100G also prevented the OS from using remaining RAM for page cache, which Cassandra relies on for SSTable I/O.
Step 4: Correlation of pauses with cluster-level symptoms
FailureDetector and MessagingService warnings were correlated against GC pause windows and nodetool metrics. During extended GC pauses the node stopped responding to gossip and message heartbeats, producing failure-detector thresholds and gossip-version anomalies; these transient states explain the observed quorum write timeouts even when nodes were reported UP after the pauses. TP stats showed some dropped range-slice messages consistent with short availability gaps.
Step 5: Recommendation, client feedback, and risk assessment
Recommendations were drafted prioritizing immediate risk mitigation: reduce heap to a safe CMS size, adjust NewGen proportion, and relax premature promotion. Client reviewed and proposed applying smaller heap settings and occupancy tuning, and asked about moving memtables off-heap and keeping stack size unchanged. The investigation made clear that JVM heap sizing and tenuring policy were the highest-risk items requiring a restart to apply changes.
Step 6: Implemented change (transition to Solution)
After agreement, the JVM options in cassandra-env were updated and nodes were restarted in a controlled manner; the implemented changes are summarized below.
Solution:
Apache Cassandra nodes were reconfigured to use a 16 GB heap (Xms16G -Xmx16G) with NewGen set to 4 GB (-Xmn4G), CMSInitiatingOccupancyFraction lowered to 65, and MaxTenuringThreshold increased (from 1 to 8). The CMSWaitDuration setting was removed to avoid forced remark timeouts, and thread stack size was left at a conservative value with a recommendation to consider 512k if stack-related errors appear. Nodes were restarted sequentially to preserve cluster availability.
These changes reduce Old Generation pressure and shorten stop-the-world pauses, restore room for Linux page cache for SSTable reads, and prevent aggressive premature promotion that previously filled Old Gen. The result is fewer long GC-induced unresponsive windows that had caused gossip/failure-detector anomalies and quorum write timeouts.
Conclusion:
Post-change monitoring showed a marked reduction in GC pause durations, disappearance of failure-detector and messaging warnings correlated with GC, and elimination of the observed quorum write timeouts. The cluster returned to stable write latency and improved operational resilience; the configuration changes mitigate the risk of repeat incidents tied to very large CMS heaps on this Cassandra release.