Designing and Delivering An Award Winning Scenario Testing Programme For a UK Bank

The UK Financial Regulator’s Operational Resilience Policy was released in March 2020 and came into force a year later. The Policy required regulated entities to define Important Business Services (IBS), map the critical assets (people, property, technology, data and third parties) that support the IBS, define the point at which intolerable harm to the customer, firm and/ or market materialised from an IBS being unavailable (also known as Impact Tolerances or ITOL) and then use scenarios to test the ability of the IBS to remain within the ITOL. Firms were expected to remediate any scenarios that exceeded ITOL by March 2025. 

For many organisations, they have already done a set of regulatory or policy-driven testing, e.g. Internal Capital Adequacy Assessment Process (ICAAP), Business Continuity Plan (BCP) and Disaster Recovery Plan (DRP) testing, which could be used as key inputs into Operational Resilience testing. The key difference was the use of severe but plausible scenarios, which often were more severe than traditional BCP and DRP tests, and were specific to IBS – a relatively new concept for organisations. 

As the UK Financial Regulator adopted principle-based regulation, there was no stipulated methodology for firms to follow. Whilst industry groups such as the Operational Resilience Collaboration Group (ORCG) were established to generate best practice, this was not available until January 2022 when the first version of [interim] guidance was published. Therefore, our client was keen to consult on different approaches they could adopt for scenario testing in 2020.

Beyond Blue defined and ran the client’s scenario testing function from 2020 through to 2024, at which point they transitioned the function from programme into BAU, and took on an advisory role for the team. From 2020, Beyond Blue identified the initial methodology, which focused on critical IBS assets and IBS-specific testing to help identify thematic vulnerabilities. The methodology evolved over the four years in several ways: 

• Detail and Depth: Over the four years, as the client matured their understanding of operational resilience and scenario testing, the level of detail we were able to capture and include in assessments dramatically increased. In 2020, most of the evidence that informed the end-to-end response and recovery timeline for any scenario was SME judgment. This matured to include isolated technical testing in test environments, which increased confidence in the accuracy of the assessments. The end-to-end response and recovery timeline also became much more granular as the understanding of technical recovery and business response increased. 
• Critical Asset Response and Recovery Strategies: A key lesson from 2020 and 2021 was that the extensive scenario library contained a set of causal events that could be filtered down to a smaller set of response and recovery strategies that asset owners would deploy. The team focused on developing a maturity model to rate the evidence available from asset owners for each of the response and recovery strategies. This was used to identify gaps in asset resilience, general capability to deploy response and recovery strategies across the organisation, but also as a key input into scenario testing.  
• SBP Metrics: One of the key issues we encountered when first engaging stakeholders was a lack of buy-in to the severity and plausibility of the scenarios we were proposing to test, as this level of scenario had not been previously tested before and went against the risk focused culture engrained within the client. The team defined a scale and provided a robust methodology to defend scenario selection, but over time additional benefits were realised including to help compare scenarios being tested for different purposes e.g. BCP, DRP ICAAP, Operational Continuity in Resolution (OCIR). The metrics also allowed the Operational Resilience team to defend scenarios that they chose not to test, or scenarios that were tested and exceed ITOL, but had no appetite to remediate. 
• Evidence and Proving Scale: One of the key regulatory focuses beyond 2025 will be firms maturing the efficacy of their testing and moving towards “simulations”; testing in production or production-like environments for multiple IBS critical technology assets and over time, whole end-to-end IBS. To help the client understand both their current level of testing and also inform their decision on how mature they want to be for different scenarios, a maturity scale was created, against which every scenario test will be assessed. 
• Integration with other Regulatory and Policy-driven testing: Organisations conduct a variety of scenario testing, exercising and assessments, often in silos, drawing on different inputs and engaging different stakeholders. This can often lead to similar scenarios, tests/ exercises/ assessments having different outputs and different results. Operational Resilience scenarios have the opportunity to bridge the gap between internal policy-driven testing (e.g. BCP and DRP) and other regulatory testing (e.g. ICAAP and OCIR) to drive efficiencies, consistency and understanding across the organisation. Collaborative scenario tests were piloted, using scenarios that could be increased in severity through a set of complicating factors to meet the objectives of a range of use cases.  

Over the period, Beyond Blue helped the client complete over 200 scenario tests across their IBS estate, identifying a series of vulnerabilities that informed a comprehensive remediation programme. The scenario testing outputs helped the client to identify key areas for investment, raise awareness of Operational Resilience across the organisation and promote a culture beyond just a traditional mindset, focusing on investment in preventative controls to one that understood the importance of response and recovery capability for worst-case scenarios. 

The internal scenario testing methodology was used as the basis for the collaborative testing with critical third parties, pre-empting the Critical Third-Party Policy (finalised in 2024). 

As part of the transition into BAU, the BAU team shadowed and worked with the Beyond Blue team for a transitionary period to help upskill them and hand over key stakeholder relationships built up over the previous four years. The Beyond Blue team also produced a large set of templates, including role profiles, RACIs, training and awareness material, scenario selection guidance, report templates, as well as a comprehensive methodology document providing guidance and insight based on their experience, for every stage of the process.

Beyond Blue continues to play an advisory role to the BAU team, but this is being phased out as the BAU team has full ownership of the process and is adequately skilled to not only complete scenario testing using the current methodology but also evolve it based on lessons learnt, industry best practice and regulatory requirements. 

“Beyond Blue quickly established themselves in managing and running the scenario testing (ST) workstream within the [Operational Resilience] programme. In order to agree the approach to ST they consulted significant parts of the organisation, bringing in external insight they had from their other relationships/contacts. A diverse and comprehensive list of STs was agreed and undertaken with documented outputs being used for senior exec syndication and sign off, which has driven thinking and ensured focus on resilience outcomes for the Group. BB has successfully conducted many senior executive briefings and to the regulator. They are fully embedded within the programme and have ensured that there is join up between ST and other parts of the programme. They have also been extremely flexible and have adapted to the changing focus of the programme and our needs. The strong foundations, seen as industry-leading, built by this team, have put us in a great position to ensure not only regulatory compliance but these foundations will be integral to building a stronger Ops Res culture within the Group. In doing so, we have developed significant relationships and trust with BB that we expect will last for many years.”