Five Signals Your Cassandra Cluster Is in Trouble

Cassandra rarely fails loudly. It degrades. By the time someone opens a ticket, the cluster has usually been signalling trouble for weeks — in metrics nobody was watching closely enough. Across health checks on dozens of large production clusters, the same five signals come up again and again.

1. Read latency is climbing while write latency stays flat

Cassandra's write path is built to be fast and it tends to stay fast. So when your p99 read latency drifts upward while writes hold steady, the problem is almost certainly on the read path — not capacity in general.

The usual culprits are tombstones, oversized partitions, or a query pattern the data model was never designed to serve. Start with tombstone warnings in the logs, then look at nodetool tablehistograms for partition size and SSTables-per-read. A read that touches many SSTables, or scans through thousands of tombstones to return a handful of live rows, is doing far more work than it appears to be.

2. Pending compactions keep growing

Compaction that never catches up is a compounding problem: more SSTables per read means worse latency, and unmerged data means disk fills faster than anyone forecast.

The cause is usually one of three things — a compaction strategy that doesn't match the workload (size-tiered on a time-series table where TWCS belongs), compaction throughput throttled too conservatively, or storage that simply can't keep up with the write volume. Watch nodetool compactionstats over time; the trend tells you far more than any single snapshot.

3. The JVM is pausing

Long garbage-collection pauses masquerade as network problems. You'll see timeouts, dropped messages, and nodes that appear to flap in and out of the cluster — and you'll go hunting in the wrong place.

Heap pressure in Cassandra usually traces back to reading too much at once: wide partitions, oversized batches, unbounded IN queries, or page sizes set too high. Check GC logs for pause frequency and duration, then correlate against dropped messages in nodetool tpstats. Collector tuning buys you room, but the durable fix is almost always in the data model or the query.

4. Load isn't even across nodes

Cassandra's whole scaling promise rests on data and traffic distributing evenly. The moment one node holds noticeably more data or serves noticeably more requests, that node becomes the ceiling for the entire cluster — and adding hardware won't help.

Look for skew in nodetool status and in per-node request rates. Causes are typically a partition key with poor cardinality or real-world skew (the classic "hot partition"), an uneven token distribution, or an application connecting unevenly across the cluster. Worth noting: very high vnode counts hurt availability on large clusters more than they help balance — a smaller num_tokens is generally the better trade at scale.

5. Repairs aren't finishing — or aren't running at all

This is the quiet one, and the most dangerous. Without successful anti-entropy repair completing inside gc_grace_seconds, deleted data can resurrect and replicas drift apart. Teams usually don't skip repair on purpose: it ran slowly, or it failed once, someone disabled it to get through a busy period, and it never came back.

Check when repair last completed successfully on every keyspace — not when it was last started. Those are very different questions, and only one of them matters.

What these add up to

Individually, each of these is a tuning problem with a tuning answer. Seen together, they almost always point at something bigger: a data model or capacity assumption that quietly stopped being true as the workload grew. That's an architecture conversation, not a config change — and catching it before a peak-traffic weekend is considerably cheaper than catching it during one.

---

RhinoSource is DataStax's trusted expert for Apache Cassandra and DataStax Enterprise, brought in to review, tune and migrate some of the largest Cassandra deployments in production. If your cluster is showing any of these signals, let's talk.