Facilitating ongoing monitoring in Model Risk Management


Verify that your model is performing as intended


A banking institution’s model governance structure is designed to provide comfort that models comply with applicable laws and regulations, model data has quality and relevance, and conceptual soundness has been validated. With these processes for managing model risk in place, it is easy to overlook another crucial element of Model Risk Management (MRM): ongoing monitoring. The ongoing monitoring process assesses whether the model is performing as intended. In addition, a key objective of ongoing monitoring is verifying an organization’s ongoing compliance with any changes to the regulatory requirements for its models.


Weakness in an organization’s ongoing monitoring of its models can lead to significant financial and reputational consequences. One of the largest U.S. banks suffered a $2 billion loss in 2012 due to its implementation of a new Value-at-Risk model that was later deemed inadequate. The new Value-at-Risk model masked losses by artificially depressing the potential risks the bank was exposed to under specific market risk conditions. Proper surveillance of model outputs and auditing of the trading strategy’s effects on the model would have provided more transparency for decision-makers and may have prevented the losses.


Successful MRM requires several processes to operate at optimum capacity. Ongoing monitoring establishes a framework that perpetually surveys the areas of model governance, data, performance and conceptual soundness. This practice allows for timely discovery of issues, necessary adjustments of models, and compliance with both internal policies and external regulation. It is important to establish ongoing monitoring during the model’s inception and throughout its life cycle. According to the Federal Reserve’s Supervisory Guidance on Model Risk Management (SR 11-7), it is essential to evaluate whether changes in products, exposures, activities, clients or market conditions necessitate adjustment, redevelopment or replacement of the model and to verify that any extension of the model usage beyond its original scope is valid.




4 components of ongoing monitoring


Firms are obligated now more than ever to ensure their MRM function is robust and that they have established effective and efficient ongoing monitoring routines. 


Case study: Evaluating model performance

After one bank faced scrutiny from its external auditor to implement proper ongoing monitoring of its credit models, it engaged Grant Thornton’s Risk Advisory practice to develop ongoing monitoring frameworks, which included key metrics to evaluate model performance and data quality. Grant Thornton implemented these frameworks in SAS and developed automated programs for the bank to use on an ongoing basis. The deliverables provided the external auditor with confidence in each model’s output, resulting in sign-off on all deliverables provided by Grant Thornton.

Ongoing monitoring can seem broad, and it may be difficult to pinpoint all that should be encompassed. SR 11-7 outlines what should generally be included in an ongoing monitoring framework. The Office of the Comptroller of the Currency (OCC) also includes key elements of ongoing monitoring as part of its Supervisory Guidance on Model Risk Management (OCC 2011-12). A comprehensive framework must include the catalogue of models being monitored and how each is measured, frequency of monitoring, risk thresholds and escalation governance. Although ongoing monitoring processes are evolving, the following four components serve as a foundation and should be embedded in every framework to ensure optimization: 


1. Inputs


In accordance with SR 11-7, ongoing monitoring includes verifying that internal and external data inputs continue to be accurate, complete, consistent with model purpose and design, and of the highest quality available. There should be active monitoring of data changes such as data type, vendor and repositories. It is imperative that these changes be tracked and documented as part of effective governance practices. Lack of monitoring for changes in data can give a distorted view of the origins of a model’s success and where improvements could be made.  


To maintain successful governance of model risk, key risk indicators (KRIs) must be embedded into the overall MRM framework. KRIs allow a firm to measure the probability of an event that will adversely impact the organization as well as measure what the consequences of the event will be. For example, the population stability index (PSI) is a KRI that measures how much data has shifted over time. As time goes on and data populations shift, outputs can become inaccurate. Firms can be proactive by using PSI scores to determine shifts in data population and deciding if adjustments are required.


2. Outputs


Ongoing assessment of model key performance indicators (KPIs) allows organizations to obtain insight on underperforming measurements, outdated measurements, and relevance of measurements. KPIs must be adjusted to reflect current and forecasted market conditions, customer behaviors, product offerings and more. KPIs should be frequently compared to the model’s goals and evaluated for alignment. By comparing predicted to actual results, a firm can better discern whether the model is performing as intended. Additionally, ongoing scrutiny of risk measurements may limit model malfunctions and in turn mitigate model risk. 



3. Model risk assessment and regulatory requirements


Model risk assessments are used to identify risks for individual models as well as the aggregate risks posed by all the models housed in the model inventory. Model risk assessments should be in sync with the organization’s model validation procedures, both in cadence and in the risk ratings for the models. The higher the risk rating for a model, the more frequent and detailed the monitoring efforts should be.


With the emergence of the Federal Reserve’s Supervisory Guidance, MRM has become a focal point for regulators and organizations. Adherence to the holistic MRM regulations and guidance is as important as the business-specific regulatory requirements applicable to specific products.



4. Reporting


Every organization should have a framework in place that contains provisions on how to respond to and escalate model errors, risks and other issues. A clear framework of roles and responsibilities in the escalation structure, reporting cadence and report components prevents ambiguity when adverse events surface. Escalation should lead up to the board to make timely, informed decisions. There is no one-size-fits-all approach to an escalation framework; however, senior management and the board should constantly be kept informed of errors, risks and other issues associated with models and the MRM function in totality. 




Maintain alignment


Governance structures, data quality, and conceptual soundness all rely on frequent monitoring and reporting to enable an organization’s models to run as intended. A faulty ongoing monitoring function can lead to losses, fines and reputational damage. Ongoing monitoring helps an organization maintain alignment with regulations, market conditions, model targets and risks. An ongoing monitoring framework must be established during the model’s inception and actively managed. In addition, findings from ongoing monitoring should be reported to management and escalated up to the board.


Throughout this series we have illustrated that risks lie not only with the models being used in financial firms today, but also in the processes meant to govern those models. Having strong governance, including robust ongoing monitoring, allows transparency into the workings of institutional models. This allows leaders to be better educated on and more aware of changing risk factors.




Our banking featured industry insights