GE Aerospace DataStax Enterprise Upgrade, Migration and Hardening

GE Aerospace is one of the world's leading aerospace manufacturers.

Its test data is captured in DataStax Enterprise, using Cassandra for the write path and Solr for search, with data from multiple sites consolidated through DataStax Advanced Replication. It is a long-lived estate: clusters that have been in production across several DSE major versions, on hardware that has been replaced underneath them more than once.

That combination — always-on ingestion, a search workload alongside it, and replication tying sites together — makes routine maintenance the hard part. An upgrade or a hardware refresh is not risky because the procedure is unknown; it is risky because the cluster cannot stop while it happens.

DataStax brought RhinoSource into GE Aerospace to make that maintenance routine — to write the upgrade and data-centre migration procedures, resolve the operational issues underneath them, and assist with security hardening.

Upgrade and migration runbooks

RhinoSource authored the production runbooks for the estate's DataStax Enterprise upgrades, including the DSE 6.8 upgrade path, so that each step had a rehearsed sequence and a defined rollback rather than being reasoned out during a maintenance window.

We designed the data-centre migration procedure around Cassandra's own replication rather than around bulk copy. A new logical data centre is stood up alongside the existing one, populated through native replication, and the old one decommissioned once it is no longer serving reads. The same procedure covers a hardware refresh inside one physical site, a move to a different region, segregating analytics onto their own nodes, and a migration to the cloud — which is why it was worth writing once, carefully.

The runbook is explicit about the failure that catches teams out: client applications connecting to the new data centre before it holds data. It specifies the driver policy and the consistency-level changes — QUORUM to LOCAL_QUORUM, ONE to LOCAL_ONE — that have to be in place before the new nodes appear, because the symptom otherwise is inconsistent reads rather than an obvious error.

We also documented the node-replacement procedure used to move onto new hardware in place, so that individual nodes could be cycled without the cluster losing a replica.

Diagnostics that could leave the building

Support cases on a cluster like this need diagnostic bundles, and diagnostic bundles carry configuration, topology and identifiers that a customer may not be willing to send outside its own network.

RhinoSource delivered a Python utility that scrubs a diagnostic collection before it is shared, so that support could be given what it needed to reproduce a problem without the customer having to choose between resolving the issue and holding onto its own operational detail.

Backup, audit and replication issues

We resolved OpsCenter backup and audit failures on the estate, and worked through defects in Advanced Replication — including a control-connection failure that appeared once the number of configured replication destinations grew past what the deployment had previously exercised. That class of problem only surfaces at a scale most installations never reach, which is what makes it expensive to diagnose from scratch.

RhinoSource also assisted GE Aerospace with security hardening of the DataStax Enterprise environment.

Why it mattered

None of this work is glamorous. It is upgrade paths, replication topology, backup verification and node replacement — the maintenance that decides whether a platform is still trustworthy in its fifth year.

The value of writing it down was that the next upgrade, the next hardware refresh and the next site migration all became procedures the customer's own team could execute, rather than engagements.