My view on CICD, branching model and deployment environments.

TL;DR

Envs:

  • staging (*) -> prod
    • (*) where CICD is done.

Branches:

  • temp: feature*, fix*, refactor*
  • perm: main

Design

What is the first env where you get automatic deployment, and the CICD pipeline runs acceptance or E2E tests against? Make it staging. That is, the env where you can prove in a PR that the CICD pipeline is green, before merging (ensuring there is no regression; and that changes are correct, assuming you wrote tests for them).

Make feature*, fix*, refactor* (or similar naming convention) temporary branches be automatically picked up by CICD (and hence deployed to staging). Delete them once PR is merged, for cleanup.

Require test evidence in PRs, including a green CICD run (proving the solution works in staging). That gives great confidence for a reviewer. Actually, it’s the single piece of information in a PR that gives the greatest amount of confidence, hence the most valuable.

The only permanent branch is main. That removes any ambiguity for developers getting started with the repository.

Code in main is releasable at any time. It’s up to you when to deploy to prod, this works well in my experience:

  • Routinely deploy to prod. For instance, each sprint or every 2 weeks. Include whatever was completed since the previous deployment.
  • Deploy to prod on demand, i.e. when a bugfix is ready or a new feature is eagerly desired.

Deploy to prod “at the press of a button”: for instance, clicking on a pipeline or adding a specific tag (eg: release/*). Whatever the method, ensure the job build # and commit id are linked in the process, for traceability.

So, the normal delivery route for changes to go live will be:

  • staging (*) -> prod
    • (*) where CICD is done.

Principles

The above practice relies on broader principles applied across the team and SDLC:

Considerations

Number of envs

The number of environments exponentially affects maintainability. Many branches and environments add complexity and little value. Make it as simple as possible. In my experience:

  • 2 envs is complex, but manageable.
  • 3 envs is hellish.
  • 4+ envs is all-out nuclear war; cannot be won, must never be fought.

Criticism

Possible concerns:

Having feature*, fix* branches deploy to stage will cause stomping between developers (as each will deploy their own changes from different branches)

Note that, an alternative like this would not solve it:

  • dev (*) -> stage -> prod
    • (*) where CICD is done; the stomping happens on dev, so no gain regarding this; and added complexity.

Doing small, incremental PRs helps (merging frequently and updating from main). See trunk-based development overview.

I would need a stable environment for UAT

Try to avoid that. If you have a stable env for UAT, unapplied changes will accumulate in batches, that when applied will have a greater probability of adding bugs. Those bugs will be harder to detect.

Go for doing UAT in staging, assuming changes are applied as soon as developers think they’re good to go. Automated tests (if the test pyramid is well applied) will prevent most issues. Other issues will be detected and reported early by PO or product team members doing UAT; great, a potential bug identified before going to prod, where it can quickly and safely fixed!

Evolution

If the highest testing and process standards are reached, then implementing automatic, continuous deployment to prod can be considered as a next step. That is, once a change is acceptance-tested in staging, immediately release it.

I haven’t been there yet, but remains as a goal for the future.

Side note

2 years since the last blog entry. That’s life… or lack of it ;)