Change management: approve changes by risk level

IT Operations · 4 min

Articles by Luís Carvalho

Treating all changes the same way creates queues and larger batches. Learn how to classify changes by risk, pre-authorise repetitive ones and generate evidence without relying on a weekly meeting.

Change management: approve changes by risk level

TL;DR

The change advisory board has become a waiting point

In many organisations, any change to infrastructure or applications goes through the same path: a formal request, a weekly meeting of an approval committee (CAB, Change Advisory Board) and a defined deployment window. The model aims to protect production systems, but not all changes carry the same level of risk. Renewing a certificate, applying a previously tested configuration change and migrating a critical database may call for different assessment and approval procedures. When they all follow the same path, requests pile up, waiting times increase and deployments may end up grouped into larger batches. For those managing IT operations, the challenge is to match controls to the criticality and impact of each change, without compromising traceability.

More formal approval does not mean fewer failures

The DORA (DevOps Research and Assessment) research programme, not to be confused with the European digital operational resilience regulation, examined this relationship in its 2019 State of DevOps report. The research identified negative effects of heavyweight external approval processes on software delivery performance, without showing that they led to a lower change failure rate. As an alternative, it recommends bringing review closer to the teams doing the work, using peer review and the automation of tests, checks and monitoring. These findings do not mean that change advisory boards are dispensable: they remain useful for coordination between teams, for communication and for decisions that require an assessment of business impact. Applying these practices to infrastructure changes should take into account each organisation's risk and control requirements.

Classify changes by risk before approving them

A common approach is to distinguish three types of change. Standard changes are repetitive, low-risk and well documented, such as certain previously tested configurations, and can be pre-authorised as long as they follow an approved procedure and are recorded. Normal changes require assessment and authorisation proportionate to the risk, ideally by people who know the systems and their dependencies. Emergency changes follow an expedited path, with the authorisations and post-implementation review set out in the applicable procedure. Classification criteria should be explicit and consider the affected services, ease of rollback, exposure of sensitive data and failure history. An up-to-date CMDB helps identify other systems that may be affected by a seemingly isolated change.

The process itself can generate the evidence

One of the reasons for maintaining formal approval processes is the need to demonstrate control to auditors, customers or regulators. DORA research indicates that peer review, supported by automation, can contribute to a reliable delivery process without relying exclusively on external approvals. In practice, each change should be linked to a request, the person responsible, the applicable authorisations, the test results and the deployment record. Service management platforms, automation tools and infrastructure-as-code mechanisms can produce part of this evidence automatically, provided that the records are reliable and access is properly controlled. When part of the operation is outsourced to third parties, for example under an infrastructure management contract, changes carried out by the provider must be part of the change management process agreed with the customer. The specific evidence and segregation-of-duties requirements should be validated against the applicable obligations.

Measure the waiting time and outcome of each change

DORA research recommends analysing the proportion of changes that require manual approval before going into production, distinguishing them by risk profile, as well as the waiting time associated with external approvals. These indicators should be analysed together with the frequency of changes that cause incidents, the need for rollbacks and the recovery time. If low-risk changes wait several days for approval, without the process adding proportionate value, there may be room for simplification. Conversely, if a category shows frequent failures, the response may be to strengthen testing, improve deployment procedures or review rollback mechanisms, rather than automatically adding another approval level. Good service observability makes it possible to quickly detect degradations after a change and to support data-driven rollback decisions.

A proportionate process requires discipline and continuous review

Approving changes according to their risk level can help reduce waiting times, avoid deployments grouped into larger batches and focus human attention on the decisions with the greatest impact. The transition depends, however, on conditions that not every organisation has in place from the start: a reliable inventory, documented procedures, adequate testing, rollback mechanisms and consistent records. Without these foundations, pre-authorising changes may simply shift the risk to operations. A gradual evolution, starting with the most repetitive changes and periodically reviewing their classification based on results, tends to be more prudent than overhauling the entire process at once. The balance between agility, security and control should be assessed case by case, depending on the criticality of the services, contractual responsibilities and applicable obligations.

Related

References

  1. DORA – Streamlining change approval
  2. DORA – 2019 Accelerate State of DevOps Report