Scaling GitOps Across 34 Kubernetes Clusters: What the Control Plane Required

At 34 Kubernetes clusters, the hard part of GitOps isn’t getting a controller to apply configuration. It’s making every change predictable across a fleet, keeping credentials inside the right boundaries and ensuring a control plane problem doesn’t take workloads down with it. A setup at Lucid Motors offers a useful view of what that takes in practice.

One Argo CD instance managed clusters across 22 AWS accounts and one Oracle Cloud tenancy, spanning five regions on three continents. A single repository held 174 Helm charts, which rendered 1,214 Argo CD Application objects. The scale changes the cost of ordinary mistakes: a values edit that works for one cluster can still break many others.

The timing also highlights a dependency teams should plan around. On October 6, 2026, Red Hat published a walkthrough of using Argo CD ApplicationSets to provision hosted control planes from Git. The next day, GitHub reported an outage affecting Actions, Git operations and pull requests, following smaller incidents on October 5 and 6. GitOps can now manage cluster infrastructure, but its delivery path still depends on services that can become unavailable.

Separate the hub’s authority from each cluster’s trust

The central Argo CD instance handled platform components and held the cluster registrations, the Cluster API controller identity and three ApplicationSets. One applied a baseline to each cluster through an app-of-apps chart. Another deployed shared platform services to the clusters hosting them. The third managed the Cluster API definitions that described the clusters.

Each spoke cluster retained control over what it trusted. Its own authentication configuration determined whether to accept the hub’s role. It also owned the role Cluster API used inside it, while workloads obtained cloud roles through the spoke’s identity provider.

That boundary mattered most for credentials. In AWS, the hub used short-lived credentials and held no long-lived cross-account secrets. Each access grant was declared on the spoke side and reviewed in Git. Oracle Cloud did not support the same arrangement: registering those clusters still required a stored credential read from a vault. That exception is a practical reminder to document where a multi-cloud security model stops applying.

Make the cluster contract the source of derived settings

The fleet became easier to operate when each cluster was represented by a configuration file with roughly 20 keys. Those fields described identity, account boundary, region, environment, API access, delegated DNS and certificates, and any opt-outs. The rest of the configuration was derived from that contract: Applications to deploy, IAM role names, Cluster API identity, DNS delegation and cost tags.

That approach reduced the work needed to change fleet membership. Adding a cluster meant changing a label on its registration; adding a component meant adding one line to a list. Matrix generators expanded those declarations across the matching clusters. A new Oracle Cloud tenancy, added for a Middle East expansion, used the same contract instead of becoming a one-off configuration. That is a meaningful test: a contract only helps if a different provider can fit it without bypassing the model.

Review rendered changes across the fleet

With 174 charts, the main operational risk was a change that rendered correctly in isolation but produced bad manifests for some clusters. CI therefore rendered the full configuration graph on every merge request. It checked each chart on its own, each app-of-apps chart against every cluster contract, every cluster definition, and each chart referenced by an Application with the exact values it would receive.

CI also rendered the target branch and posted a manifest diff to the merge request. That gave reviewers a view of the resources the change would alter, rather than asking them to infer the result from template edits. For large fleets, rendered output is the useful review surface: it shows how a single source change fans out.

Interpret the operational measurements with care

Reported out-of-band drift fell from 24.3 incidents a month before cutover to 0.9 afterward, measured across two 90-day periods. Those figures came from different sources. The earlier count used incident tickets tagged as drift, and tagging was incomplete; the later count came from controller diff events with a confirmed out-of-band cause. They do not establish a clean 96% reduction.

The more defensible finding is that once drift was instrumented, confirmed cases were rare, and the controller corrected each within its reconciliation interval without manual intervention. Teams evaluating a similar change should define a consistent way to count drift before and after adoption.

Limit reconciliation’s blast radius

A bad commit was the failure mode the team most wanted to contain. Every ApplicationSet preserved deployed resources when its entry was removed, preventing deletion from cascading through the fleet. Cluster objects also carried a no-prune setting, so reverting a commit could not remove a running cluster. These safeguards made deletion harder to trigger through routine reconciliation.

The same design principle applies during an outage. Workloads run in the spoke clusters, so they can keep running if Git or the hub becomes unavailable. GitOps-driven deployments and configuration changes will pause until the control path returns. The desired failure mode is a delay in changing workloads, not an outage of workloads already running.

The hub also deserves the strongest protection in the system. Once a change merges, a compromised repository or CI path may have access to the hub’s reach across the fleet. Review controls and credential boundaries need to reflect that concentration of authority.

Onboard through a repeatable path

Adding a cluster involved four steps: create the matching IAM roles once in Terraform, write the cluster values file, open a merge request so CI renders the full graph and posts its diff, then merge and label the registration. Older clusters that had been built by hand could be adopted in place through an explicit import path.

Mean onboarding time was reported as 4.0 days for eight earlier clusters and 1.7 hours for eleven clusters after cutover. The first figure was reconstructed from change tickets; the second measured the time from merge to first successful sync. The cutover also removed manual approval steps, and the samples were small, so the figures aren’t a direct comparison. They do point toward a faster, more repeatable process.

The broader lesson is familiar: declare desired state, review it and let controllers reconcile it. At fleet scale, shortcuts around credentials, rendering and deletion controls become much more consequential. The cluster contract and the rendered diff make changes easier to reason about; the next test is whether those safeguards remain effective as providers and cluster counts grow.

Nitish Mane is a software engineer at Philo, where he builds infrastructure for AI-assisted production operations. He previously worked on platform infrastructure at Lucid Motors, including the control plane described here.