How ANZ is using Golden Pipelines for engineering acceleration
Kieran Goldsworthy presents ANZ’s approach to standardised CI/CD pipelines to enable fast and confident delivery
ANZ’s got oodles of Apps. Cloud portals, mobile apps, tax calculators, mainframe core, regulation compliance, data meshes, CI/CD platforms, and so on.
This blog has previously talked about the Golden Path, a way of standardising, streamlining, and uplifting the app development and deployment process. As a way of accelerating adoption of the Golden Path to the highest standards, and towards best practise, we are building Golden Pipelines.
This library of Golden Pipelines is a service build around the idea of ‘standardised pipelines’.
Why standardised pipelines are important
The primary concern of a developer is the app they are working on; how it gets built and deployed isn’t all that important to their core responsibilities.
Developers might be interested in such things – solving technical challenges is part of the fun in the job – but primarily they only need to think about them because it’s not taken care of on their behalf.
They don’t work in a vacuum though, and there remains a need to meet business and security compliance requirements (e.g. any findings in a code security scan); this should be surfaced as some form of Engineering Feedback (feedback loop).
We have all grown comfortable with VMs and higher abstractions still. Gone are the days that app developers need to consider exactly which rack in the datacentre to deploy to. Now it’s just set an ‘environment’ (e.g. Availability Zone) and you’re done.
We can see the same transformation is now happening with CI/CD. Consider that:
The developer needs to know which programming language they are writing in, but deployment automation makes the artefact’s format irrelevant.
The developer needs to know if any tests failed, but not to remember they need to be run.
The developer needs to know which environment to deploy to, but not the intricacies of networking and authentication.
Standardized Pipelines are just another level of abstraction, a service, a trusted team that has solved and automated many of the challenges for the common patterns and use cases - only feeding back information that the developers actually care about.
For many developers, this can be scary because they have a CI/CD process that ‘works’ and they have spent much time getting to that state. They feel a sense of ownership over the process and can be protective. But that will pass. Once they get a taste of a reliable pipeline being written for them where the complexity is abstracted away, they will struggle to go back.
By putting the engineering effort for building and maintaining these tools into a common, standardized, and centralised service (Golden Pipelines), we gain advantages beyond just the technical capabilities:
Documentation will have way more readers (read: ‘production testers’), and like river stones, the more it’s used the more the sharp points are smoothed out
Good help is easy to get because the knowledge is commonplace
The meta-tools will have increased engineering attention as:
> Onboarding process is automated and refined
> Compliance and reporting become feasible at scale (measure, then improve)
> Increased confidence in the technical pipeline leads to streamlined business processes (e.g. automated change management)
If the engineering effort was spread out to hundreds of tools, then you have hundreds of poor tools. Why not invest in a single, really good one?
Building ANZ’s pipeline library
We have already been building and releasing Golden Pipelines – as well as helping teams to migrate – for 6+ months now. We have focused primarily on GitHub Actions as one of our target state standard tools, so much of the technical implementation to date is driven by GitHub’s capabilities.
Our ‘library’ of Golden Pipelines uses a centralised model. By referencing a centralised pipeline, as opposed to creating a local copy, we can roll out updates automatically. Engineers need only reference the mutable Major tag to always be using the latest stable release.
We are also ‘inner-sourcing’ the pipelines. This means anyone in ANZ can contribute to the library, adding new pipelines or improving existing ones.
For the time being, we are not enforcing their use. It is our intention that they will be better than what engineers are doing themselves. But we do provide a break-glass scenario: teams that find an urgent bug, or need an urgent change, can clone the pipeline to their own repos, make the change, and reference that copy if they can’t wait for the central team to approve the fix. We only ask that those who do this contribute the fix upstream for the rest of the organisation to enjoy.
How it’s implemented
As mentioned, we are focusing on building the library of pipelines on top of GitHub Actions. I won’t dive too much into the decision to use Actions beyond referring to previous discussions on consolidation.
This manifests as a few dedicated Orgs and Repos that we use to offer and maintain the service.
The standardised pipelines are technically Reusable Workflows and are stored in a mono-repo that is accessible to the whole enterprise to read. This repo is granted special permissions to share its Actions-related resources so other pipelines (controlled by the engineering teams) can reference it.
The users will have their own pipeline, saved in their own repos and triggered based on some event. This user-owned pipeline acts as a wrapper/parent and calls the Golden Pipeline (child) as one of its jobs.
jobs:
run_golden_pipeline:
uses: tooling/gha-golden-pipelines/.github/workflows/some_golden_pipeline.yaml
with:
.
.
.
We make use of git Tags to version each pipeline independently (prefixed with the pipeline name) using Semantic Versioning. This allows users to specify exactly which version of the pipeline they want to run by specifying the tag after an @.
For example, to run the windows-specific msbuild pipeline version 1.0.1:
jobs:
run_msbuild:
uses: tooling/gha-golden-pipelines/.github/workflows/windows-msbuild-build.yaml@ windows-msbuild-build-1.0.1
with:
.
.
.
By also publishing mutable major tags, users would only need to specify that major version to always get the latest improvements with minimal disruption. In the above case, they would replace the tag with windows-msbuild-build-1.
All pipelines (both wrapper and Golden) are composed of steps, which can be blocks of scripting or a reusable set of steps packaged as an Action. This allows anyone to write a reusable script intended to run as part of a larger pipeline.
Some of these we build internally as they are specific to our environment, but others we import from the marketplace.
Being a bank, we need to prevent running untrusted code, as such, we do not allow running arbitrary actions.
Instead, we have developed a process that imports them into our own environment in a dedicated Org (1 repo per action). This includes pulling specific versions, running them through code security scanners, and publishing versions automatically modified to be compatible with our set-up.
Finally, we need to test any changes we make to the pipelines before we publish a new version. For this, we have developed a testing framework to automate the running of regression tests against the relevant pipeline in a sample codebase.
Future vision
We have onboarded a handful of application teams to date, and they are enjoying it much more than when they had to manage pipelines themselves.
‘We've had two successful go-lives and multiple test deployments so far […] successful production rollout with zero impact on delivery.’ ‘It has drawn immense benefit’ – Abhinav Kumar, Group & Wholesale Risk.
‘I was surprised how quickly I was able to get a basic workflow up and running. […] After the first application had been migrated to the golden pipeline and tested, all the remaining application builds were quickly migrated.’ – Donald Forrester, Banker Digital Experience.
Now that the concept and its value have been proven, we are about to embark on a multi-year journey to migrate the rest of the bank. As we do so, we will identify more and more use cases worthy of building a standard around; growing the Golden Pipeline Library further.
The library is just a part of our vision for the future of CI/CD at ANZ. Our driving philosophy is ‘You bring it, we build it,’ meaning app developers focus on their code, with the whole process to get changes to production is just a ‘git push’ away.
We are building this from a simple tool into a full service with:
Centrally managed runners
Workload-based ‘identity’ (removing the need for long-lived credentials)
Automatic onboarding to ancillary tooling, like security scanners and secret stores
Managed artefact storage
.. and so much more!
We are very excited to make the dream of CI/CD as a Service (will CICDaaS catch on?) a reality.
Kieran is a Senior Cloud and DevOps Engineer, working in the Continuous Delivery squad to improve the development tooling used throughout the bank. He has been specialising in CI/CD for the past six years, building and maintaining both pipelines and platforms. With experience ranging from legacy systems administration to cloud solution architecture to full-stack application development, he believes the best engineer is a well-rounded one. Learning new skills, automating everything, and leaving things better than when he found it are some of Kieran’s core philosophies.
This article contains general information only – it does not take into account your personal needs, financial circumstances and objectives, it does not constitute any offer or inducement to acquire products and services or is not an endorsement of any products and services. Any opinions or views expressed in the article may not necessarily be the opinions or views of the ANZ Group, and to the maximum extent permitted by law, the ANZ Group makes no representation and gives no warranty as to the accuracy, currency or completeness of any information contained.