CI/CD: Why You Want It, Why It Might Make You Nervous

Continuous Integration/Continuous Deployment (CI/CD) is widely promoted as best practice in a DevOps environment. By integrating granular code changes with a testing and review pipeline, CI/CD bridges the gap between developers’ desire to push new features live and operations’ preference for stable, static conditions that are easy to support.

At its best, CI/CD offers an optimal path to providing innovative features to users, while allowing an organisation to react quickly to market needs and opportunities. However, for those with caution born of experience, the idea of new code being pushed to production too easily is an uncomfortable one.

Exactly how code is approved for publication depends on circumstance. At one end of the scale, there are internal apps that don’t sit in a business-critical pathway. At the other, there are public apps that perform high-profile jobs in a highly regulated market. A lot depends on the quality of the testing process and the history of stability across the product life cycle. CI/CD is a powerful approach that needs responsible management.

Deciding on action

During an audit, one of our clients discovered that the decision on the final push to production could be – and often was – made by a single person. However, the company had changed policy to require that two people must agree to take a product live. No particular problems had led to that policy. Rather, it was a product of natural managerial caution.

The client asked us to help evolve a system that provided safeguards, without losing too much of the CI/CD benefit. The company is a large organisation with many sub-groups and a large contingent of contractors and third-party developers, with good role differentiation and access compartmentalisation.

Analysing the situation

The tradeoffs boiled down to – as they so often do – security versus utility. If the safeguards were too onerous, deployment delays could occur that would force staff to find shortcuts to do their jobs. But make them too easy and the approval process would become a rubber-stamp exercise that solved nothing except ticking an audit requirement box.

We considered the company’s structure and the experience level of the staff already involved in the CI/CD pipeline – who had been quietly pushing product live without incident. With that in mind, we were comfortable with the idea that, once a developer decided to send something to production, their peers in the pipeline would get a request for approval. This would basically be a simple alert with details and a button.

The CI/CD system used the Jenkins automation server to manage deployment. The existing deployment jobs were created purely in the Jenkins UI rather than via code, so we disabled the existing jobs and had the Jenkins deploy stage point to the existing deployment script, replacing the UI component with code.

Implementation

We couldn’t repurpose existing code because different teams had different layouts, so we took the opportunity to develop and deploy a consistent approach. We created a self-hosted GitLab central repository, within which each team had a Groovy file holding the permission definitions, such as the ability to look up the build user.

The two-key part simply asks for input during the deployment job, with “approve” and “abort” buttons. This checks who the user was that initiated the job and compares it to the user who hits the “approve” button – so these must be functionally different to progress with the release. It’s very simple and has minimal impact, yet fulfils the client’s policy.

An important part of the project was to explain this to developers and get their buy-in. They were not resistant to the extra step but did ask why – given their own development and testing protocols had produced consistently good results – it was necessary. We explained that it added a safeguard with few drawbacks. What’s more, we noted, it could head off problems in the future and add to corporate confidence in the development system. The developers accepted that this had benefits for their work.

Conclusion

It’s a good idea for organisations to periodically review their procedures and protocols, even if they haven’t had particular problems. Security and compliance work best when they have minimal impact – which can leave policies prone to stagnation. By careful appreciation of existing, successful working practices, it is possible to introduce changes that improve reliability without disrupting productivity. Conversely, doing so also helps to resist changes that would be detrimental – by demonstrating competence at change management and awareness of the implications of change at each level of an organisation.

Furthermore, modern DevOps is flexible and adaptable. This makes it easier, not harder, to implement changing corporate priorities without disrupting innovation and productivity. It is as much a mindset as a group of tools and procedures – and learning to appreciate its potential is a key organisational asset in itself.

If you want to know more about managing DevOps and security, especially in a modern, scalable, cloud-based and serverless environment, talk to us at hello@and.digital