Auditing Deployments At Scale

Geoff Lillis

The Challenge

How can you tell when you've shipped production code? Some like dashboards, some check release notes, others still prefer to subscribe to alerts. All three groups look to me in confusion when I say you can do it with baked goods.

Historically

Back in 2017, when I joined Fenergo, we were averaging three releases of our core product a year. Each new version was a struggle through merge conflicts and manual testing: time consuming events that added little in the way of client value. Success called for a celebration, and those celebrations often involved donuts. If all other markers of a new version passed a member of staff by we could depend on the presence of confectionary to alert them to the change.

We were aware that infrequent releases are an antipattern. I've heard it said here that releases are like flossing: do it daily and it's pain free, do it every couple of months and it's unpleasant.

We've learned from those times. We've improved, and these days we're making production changes at least a thousand times a month.

Demonstrating Controls

How do we demonstrate our controls to internal and external auditors? With three releases a year, manual audits are a reasonable approach. Want to track who signed off on the release? You were probably on the call. Need to know what code was involved? You've probably memorised the tag. Want screenshots of test results? No problem.

Meeting Current Velocity

Approaches that work adequately at three releases a year do not scale elegantly to our current velocity, so our Security and SRE teams worked together on getting the information we needed in an accessible fashion. One of our guiding principles is to keep our tests, controls, and checks within our pipelines. We use Azure DevOps, and the deployment record was our starting point. The rest api has a property indicating the associated pipeline run. From this we could get information about how the code was built, tested, and deployed. This led to information about the commit, and, from there, details of who reviewed the code.

The only item we didn't have was the approvers. The deployment process for our SaaS Application requires two directors to approve anything that goes to production. Within the pipeline this is tracked in an area called 'approvals and checks'. Azure DevOps does not currently have a documented rest api for this. The options seemed to be checking tens of thousands of deployments manually or admitting defeat and asking the auditors to accept a sample.

Solution was in Azure DevOps

Instead we tried watching network traffic while looking up approval information through the web interface. We noticed calls to _apis/Contribution/HierarchyQuery/project, and responses which contained the details of the release stage approvers. After that we just had to recreate the payload and we had the required information.

I confess I got too caught up in the technical side of things. An auditor asked me for screenshots of all pull requests for the previous six months in a Word document and responded with mixed enthusiasm when I told him I'd need to check Word's max page limit first. We talked it through and showed him an interface where he could quickly link to evidence of each control which let him self-serve on the evidences he required.