Designing a CI/CD pipeline that fails safe
A CI/CD pipeline has one job that matters more than speed: it must refuse to ship a broken build. Everything else — the caching, the parallel jobs, the dashboard — is secondary. If a pipeline can deploy a failing build, it will, at the worst possible moment. So I design pipelines around that one rule, and every other choice follows from it.
The first decision is to separate building from deploying. The CI job builds the code, runs the tests, scans the image, and pushes it to the registry with a digest pin — a unique hash, not a moving tag like latest. Nothing in that job touches the cluster. Deployment is a second, separate act: a change to a Git repository that says “run this digest.” The cluster, or the GitOps controller watching the repository, picks it up from there. If the build fails, there is no new digest and nothing to deploy. If the deploy fails, the build is untouched and the previous digest is still right there in Git.
The second decision is that every gate is a real gate. Tests run on every pull request, and a red build blocks the merge — no overrides, no “merge it and fix it later.” A static analysis scan runs on the code, and an image scan runs on the built image, and both must pass before the image is pushed. These gates feel slow the first week, then disappear into the background. What they replace is the 2 a.m. page that says a bad image is in production.
The third decision is about secrets. A pipeline needs credentials: registry logins, cloud keys, deployment tokens. Those belong in the CI system’s secret store, injected at run time, and nowhere else. They never appear in a log, never sit in a repository, and never get baked into an image. If a secret leaks into a log, rotate it and fix the logging. On my own projects the rule is absolute: no hardcoded passwords anywhere, in any file, for any reason.
The fourth decision is the rollback plan, written before the deploy, not after the incident. A deploy that cannot be undone is a gamble. With digest pins and Git as the source of truth, a rollback is a revert: the previous known-good digest goes back into the repository and the cluster returns to it. I also keep deployments small and frequent. A small change that breaks something is easy to find; a week of changes shipped at once is a mystery.
Finally, the pipeline should tell you when it fails, loudly. A failed job that nobody notices is worse than no pipeline at all, because it creates the illusion of safety. Failures go to a channel somebody reads, and a failed deploy blocks the next one. A pipeline that fails safe is a pipeline you can trust — and trust is the only reason to automate deployments at all.