From ad-hoc deployments to controlled multi-stage delivery

Tuesday 8:30, same symptom, different root cause

The first customer bug report is not a Kubernetes bug. It is often an artifact and promotion bug.

A company with one-time velocity asks engineering to push a hotfix. Development runs it through CI. Staging uses a modified chart overlay. Production receives a different tag because a release script was manually adjusted overnight. At 8:30, support asks why API responses differ between staging and production for the same ticket.

No one can answer it quickly with evidence: Which commit was actually promoted? Which pipeline run produced the production image? Which GitOps revision is currently synced? In teams under pressure, the first answer often becomes “let’s rebuild and redeploy.” That can work temporarily, but it destroys traceability.

This is common in companies with 10–100 people:

• high delivery ambition,
• constrained platform capacity,
• distributed responsibilities,
• growing number of services,
• and still a lot of ad-hoc release logic.

The real question is not “should we do Kubernetes?” The real question is: “how do we make delivery as auditable and repeatable as the business expects it to be?”

Why most release chaos is not a tooling issue

A lot of teams unintentionally optimize the wrong stage. They optimize for quick environment updates and skip the control layer.

Common pattern:

• each environment triggers separate builds,
• release scripts are patched during incidents,
• environment parity is approximated by hand,
• release notes exist in chat history.

The operational side effect is consistent:

• version lineage disappears,
• rollback points become uncertain,
• ownership becomes person-dependent,
• and review cycles become late and emotional.

GitLab CI/CD is designed as a release control surface with explicit pipeline stages, jobs, variables and environment logic. If those controls are ignored in delivery decisions, teams are managing symptoms rather than a delivery model ¹.

Delivery architecture before platform architecture

Kubernetes is often treated like the bottleneck. In practice, it is usually the final execution layer. The delivery control problem starts earlier.

A robust SME model has three separations:

1. Build responsibility in GitLab CI
2. Artifact custody in Nexus3
3. Reconciliation in ArgoCD

This is the practical meaning of controlled promotion: one build, deliberate promote, predictable rollback path.

The control chain in concrete terms

Step 1: GitLab as the single source of truth for release intent

Start with a strict release intent in CI:

• merge and validation rules on the source branch,
• deterministic build and chart validation stages,
• one artifact family per release candidate,
• explicit environment definitions for dev, staging, production.

For production, use deployment approvals and protected environments so promotion is a controlled action, not a terminal side effect ².

Step 2: Nexus3 as immutable artifact hub

The key operational shift is this: not “build per environment,” but “build once, promote with provenance.”

Without this shift, teams can end up with three different tags for the same logical change.

Nexus Repository supports managed artifact formats used by container and chart workflows and gives a practical place for artifact custody before deployment ³.

Step 3: ArgoCD as pull-based reconciliation, not a deployment script replacement

ArgoCD’s GitOps model is declarative and pull-based by design: desired state in Git, cluster convergence through reconciliation, and explicit sync outcomes ⁴.

For teams, this matters because it makes drift visible and provides a cleaner evidence base for questions like:

• what was promoted,
• when,
• under whose approval,
• and with which sync result.

Multi-stage release without enterprise overhead

The model remains practical if you avoid overengineering.

Development

Keep it fast and iterative. Build, test, package, publish artifact. At this stage, velocity is expected.

Staging

Treat staging as verification layer with explicit criteria:

• integration test pass,
• config validation,
• optional load and boundary checks,
• release notes in the artifact context.

Production

Restrict direct changes. Use an explicit change owner and approval path.

Typical production guardrails:

• required approver,
• no unsanctioned manual image changes,
• rollback target known before go-live,
• post-release check within an agreed window.

This is achievable for small teams because you are not adding five systems overnight; you are adding decision discipline to one path.

Helm where it belongs, Kustomize where it helps

One frequent error is to apply Helm and Kustomize inconsistently across every service. The practical split for teams with one release pod:

• Helm for application packaging and release semantics,
• Kustomize for selective structure adjustments across namespaces or cluster service surfaces.

Helm’s model is chart packaging and versioned deployment units ⁵, while Kustomize’s base/overlay approach works for scoped configuration adaptation ⁶.

What this improves for SME teams (with realistic data)

In a team with limited platform headcount, this model reduces ambiguity:

• fewer midnight rebuilds for environment parity,
• less time tracing who changed deployment inputs,
• clearer release decisions in cross-functional reviews,
• easier emergency handling because rollback has a known candidate.

A practical planning frame many teams use: if one production change is critical, aim for one approved artifact path and one sync strategy. If two unrelated paths are needed, split responsibility before production, not during an incident.

These are process guarantees, not magical metrics.

A realistic SME operating envelope helps keep the model practical: for a 40-person team with 3 developers and 1 platform-owner, 10 to 20 services usually generate 80 to 120 commits to main per month, with 4 to 8 release candidates in production per month.

If each environment currently builds independently, you are managing up to three different image candidates per logical change. The controlled model keeps one candidate and adds fixed review gates: staging pass, one owner approver, and a documented rollback target before production sync.

Operational roles and decision points

A minimal model can work with four roles:

1. Service owner — keeps pipeline health and build quality.
2. Delivery owner — decides when an environment handover is allowed.
3. Reviewer — verifies staging outcomes before production.
4. Incident owner — manages rollback and post-deploy corrections.

This keeps ownership explicit without waiting for a full platform team.

Common objections you will hear

“This is too heavy for us.”

You do not build full enterprise GitOps everywhere. You start with one workflow that is business-critical. The model is not about more tooling; it is about less ambiguity.

“Can’t we just use raw manifests?”

For tiny scopes, yes. But if you need repeatable release semantics across teams, chart-based packaging gives clearer release semantics. Kustomize overlays remain useful where structural overrides are clean and narrow.

“Will this slow deployment?”

Not if you keep production gates proportional. The goal is faster diagnosis during incidents, not more ceremony during routine dev pushes.

“How is this different from our current CI/CD setup?”

In the current setup, “deploy” is often still a mix of manual environment drift and manual promotion. In the controlled model, the artifact reference and promotion intent remain constant while environments progress from verification to production.

The 6-week implementation pattern we use in practice

Weeks 1–2: define the chain

• select one release-critical service,
• define environments and approval roles,
• standardize image/chart naming and retention notes.

Weeks 3–4: stabilize artifacts

• implement single-build per release candidate,
• publish immutable images and charts to Nexus3,
• store artifact references where both teams and auditors can trace them.

Weeks 5–6: controlled promotion

• introduce ArgoCD sync from a release repo,
• add production approval gate,
• run a rollback drill on a noncritical change.

What you can promise — and what you should not

Promise:

• clearer ownership,
• reproducible promotion path,
• faster incident triage,
• better visibility on what is actually running.

Do not promise:

• zero incidents,
• legal or compliance certification as a side effect,
• instant maturity without team rhythm.

Delivery discipline supports evidence. It does not replace operational ownership.

Footnotes

1. https://docs.gitlab.com/ci/ ↩

2. https://docs.gitlab.com/ci/environments/deployment_approvals/ ↩

3. https://help.sonatype.com/en/formats.html ↩

4. https://argo-cd.readthedocs.io/en/stable/ ↩

5. https://helm.sh/docs/v3/topics/charts/ ↩

6. https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/ ↩