Guide: Pipeline Structure Refactoring

In this guide, we are going to walk through some simple refactorings to make your build pipelines more expressive, easier to read and easier to maintain.

While these steps are in a particular order and tell a story of iterating on the pipeline, don't assume that the final result is necessarily what you should aim for in every situation. Each of the styles and refactorings here has potential benefits and drawbacks in terms of complexity, maintainability and readability. Try them for yourself and mix and match what fits well in your context.

Intro: The initial set up

We start off with a pipeline you might expect to find to test and deploy a typical web application:

(def pipeline-def
  `(
     (either
       wait-for-manual-trigger
       wait-for-commit)

     (with-repo
       run-unit-tests
       run-acceptance-tests
       build-artifact
       publish-artifact)

     check-preconditions-ci
     deploy-ci
     smoke-test-ci
     run-ci-tests

     check-preconditions-qa
     deploy-qa
     smoke-test-qa

     wait-for-manual-trigger

     check-preconditions-live
     deploy-live
     smoke-test-live

     report-live-deployment))

It waits for commits to a repository (or someone triggering it), runs a few tests and deploys to three environments: A CI environment where additional automated tests will run, a QA environments for manual, exploratory testing and, after a manual signoff, to production. Each deployment first checks if the environment is ready to be deployed to, deploys and runs a smoke test to make sure everything went well.

Step 1A: Adding some structure

The above pipeline does the job and the code doesn't look too bad. But the only thing structuring it right now is newlines in the code. Removing them, you'd probably be lost. And looking at the pipeline in the UI, you are probably lost and can't even fit a nice overview onto your screen:

So let's try adding a bit of structure to the code. The run control flow element is perfect for this: it gives things that belong together a common container but doesn't add any additional behavior.

Let's start with deployment:

(run 
       check-preconditions-ci
       deploy-ci
       smoke-test-ci
       run-ci-tests)

Ok, that looks better. But now, pipeline just looks like a set of run steps cluttered across the UI:

So we need a way to rename those steps into something more helpful. That's what the alias control flow element gives you:

(alias "deploy to CI"
  (run
    check-preconditions-ci
    deploy-ci
    smoke-test-ci
    run-ci-tests))

We can also rename some of LambdaCDs built-in steps to better fit the context they are used in:

(alias "wait for signoff"
  wait-for-manual-trigger)

Combining alias and run, we end up with a much cleaner pipeline:

Step 1B: Compacting Steps

While run and alias are great to group detailed steps, not every detail deserves its own step, clearly visible to everyone. Instead, we can use the chaining macro to create a step out of sub-steps:

(defn complete-ci-deployment [args ctx]
  (chaining args ctx
            (check-preconditions-ci injected-args injected-ctx)
            (deploy-ci injected-args injected-ctx)
            (smoke-test-ci injected-args injected-ctx)))

The behavior is still the same, but now the details of the deployment are hidden from the user of the UI, just visible in the output of the complete deployment step:

Step 2: Removing Duplication

OK, so our pipeline now looks quite a bit more clean. But did you notice we had to more or less duplicate our changes for all three deployments? This feels like duplicated code that we could clean up. I'm going to go forward with the compacted steps from step 1B but you could do the same refactoring with the approach from 1A as well.

Instead of creating separate steps for every environment, we are now trying to parameterize our steps. What we are trying to achieve is this:

(def pipeline-def
  `(
     (either
       wait-for-manual-trigger
       wait-for-commit)

     (with-repo
       run-unit-tests
       run-acceptance-tests
       build-artifact
       publish-artifact)

     (deploy :ci)
     run-ci-tests

     (deploy :qa)

     wait-for-manual-trigger

     (deploy :live)
     report-live-deployment))

For this, we need to refactor our deployment-steps, get rid of the duplicates and put in parameterized versions instead:

(defn smoke-test [args ctx env]
  (step-support/capture-output ctx
                               (println "running smoke tests against" env "environment...")
                               {:status :success}))
                               
(defn check-preconditions [args ctx env]
  (step-support/capture-output ctx
                               (println "checking preconditions to deploy to " env "environment...")
                               {:status :success}))
(defn do-deploy [args ctx env]
  (step-support/capture-output ctx
                               (println "deploying to " env "environment...")
                               {:status :success}))
(defn deploy [env]
  (fn [args ctx]
    (chaining args ctx
              (check-preconditions injected-args injected-ctx env)
              (do-deploy injected-args injected-ctx env)
              (smoke-test injected-args injected-ctx env))))

Obviously, the steps don't do anything in this example but you get the point here.

In the last step, we made big improvements on the UI side. This time, the pipeline almost didn't change at all:

However, we got rid of some of our code duplications, making our pipeline code easier to understand and to maintain.