Making model risk management a core business function

Model risk managementTo comply with stringent regulations like the Comprehensive Capital Analysis and Review (CCAR), Dodd-Frank Act Stress Test (DFAST) provisions, and OCC 2011-12/SR11-07,1 the Federal Reserve and Office of the Comptroller of the Currency Supervisory Guidance on Model Risk Management, banks must create a distinct risk management culture. Those that treat model risk as a core business function can create sustainable, efficient and cost-effective model risk
management (MRM) programs.

Constant regulatory change quickly turns what might have been nice to have in last year’s MRM framework into a must-have this year. For example, CCAR 2015 expects bank holding companies (BHCs) to provide a comprehensive inventory of models used in their capital planning projections, plus the instruction requires that the list of models should be organized around reporting form FR Y-14A, Capital Assessments and Stress Testing, which is new — last year, banks did not have to organize the model submission in alignment with FY 14A. At the same time, banks need the appropriate resources to manage model risk effectively. To evaluate additional resource needs, banks must conduct an assessment that helps them understand the gaps in their ability to meet their MRM requirements.

In addition, the use of financial models carries inherent and residual risks that need to be appropriately managed, mitigated and measured. To accomplish this, banks need to establish a monitoring process to accurately measure model risk on an ongoing basis, including identifying and developing appropriate key risk indicators (KRI) at the individual model level and in the aggregate.

Ongoing monitoring metrics of model risk at aggregate level can be grouped into three categories: financial, quality, and time/frequency. Examples of performance indicators to measure and monitor model risk at the aggregate level include:
  • Number of models with exceptions
  • Number of matters requiring attention/matters requiring immediate attention (MRIA)/audit findings
  • Number of findings based on materiality/severity
  • Number of MRIA due to poor data quality
  • Frequency of model validation
  • Frequency of ongoing monitoring
  • Number of validators per model

Inherent risk of individual models can be measured based on model materiality, complexity and potential business impact. Inherent risk can be mitigated using MRM controls through the three lines of defense. For example, model developers should consider the limitations of the model based on the choice of model development methodology. During model validation, the second line of defense should review the conceptual soundness of the model and compare the methodology against alternative theories.

Ongoing performance monitoring to measure residual risk for individual models can be grouped into performance, stability and diagnostics metrics. Examples of KRIs to measure and monitor residual model risk for individual models are:

  • Mean percentage errors
  • Coefficient stability
  • P-value

chart 1

Ongoing monitoring
Firms with an optimized MRM function aim to develop comprehensively appropriate performance indicators:

  • At the model level to manage the individual model risk
  • At the aggregate level to manage the health of the model inventory

The SR11-07 regulatory guidance identifies ongoing monitoring of model performance as a core element of the validation process.

As many banks engage in necessary MRM activities, they are recognizing that the need for effective MRM goes beyond simply meeting basic compliance requirements. In practice, implementing a robust and sustainable MRM framework is a complex and resource-intensive organizational endeavor that requires coordination across multiple functions. The optimal result requires a cultural change and is nothing less than an organization-wide integration of MRM.

chart 2Evolving challenges
An effective MRM program must address numerous evolving challenges. Despite the fact that the processes and governance related to MRM are complex, the processes banks actually have in place are often manual and reactive, with insufficient operations oversight.

When using different model methodologies across their organizations, for example, it is very common for banks to use loss forecasting model methodologies across risk, finance and lines of business that are disjointed, producing conflicting results that create a healthy tension, which must be reconciled and properly overlaid across their MRM frameworks. Further adding to the complexity is the fact that it is necessary to integrate the top-down model methodologies typically used by banks subject to DFAST with the bottom-up methodologies employed by larger banks subject to CCAR. In addition, model validation inconsistencies are commonplace.

Regulators are seeking hundreds of pages of documentation per model, leaving banks wondering how they can possibly meet those requirements in any kind of sustainable fashion. Models from various areas of a bank can have different levels of complexity and granularity, necessitating communications between its risk, finance and treasury areas and its lines of business to report appropriately on model risks.

Often, if a bank has an enterprise function for model validation, lines of business rely on the enterprise model validation function to perform all the model validation activities. However, the first line of defense in the line of business is equally responsible for conducting model validation during the model development. This is not an advisable strategy, because it overtaxes that group and deprives the rest of the bank of the opportunity to become a vested part of the solution.

The quantity and quality of data is a further challenge. Small banks might not have enough data granularity, and large banks might have too much data. A model is only as good as the quality of the data used. Some banks have been improving their modeling techniques; however, the level of data granularity still does not always support the complexity and number of model variables. In addition, smaller banks merging with larger banks might bring too little data to support accurate forecasts.

“The goal should be to generate enough reliable data to support forecasts over at least nine quarters.” — Ilieva Ageenko, Leader, Grant Thornton Model Risk ManagementFinding the necessary resources
An important element for banks to consider in crafting an effective MRM framework is determining the combination of resources that best supports their model validation and documentation efforts. Most banks usually don’t have adequate in-house resources for all the required MRM tasks and must outsource some or all of the process.

An additional consideration in outsourcing MRM tasks is that the constant changes in the regulatory environment make it important to use resources that have an understanding of how regulations and requirements are changing, not just the technical know-how to assist with testing and documentation.

The three lines of defense in the MRM framework are:

  1. Model developers
  2. Model validators
  3. Internal audit

Large banks typically handle model development inhouse, and some do model validation in-house as well. Smaller banks usually outsource model development and validation to outside consultants.

Internal audit’s role as a line of defense presents additional challenges. Internal audit should review the entire model development life cycle as well as model validation, but many banks lack the resources to do so and outsource some of those responsibilities. In the case of small banks, internal audit often isn’t looking at the MRM framework at all.

Managing the business of model risk
To truly manage the business of model risk, a bank must create a sustainable and systemic MRM infrastructure. This includes developing appropriate and consistent indicators to measure model risk function performance. It also includes weighing various resource alternatives to find the best, most costeffective mix when assembling the MRM framework. To aid in measuring model risk function performance, we recommend developing a dashboard that visually monitors key performance indicators (KPIs).

Bank MRM functions typically develop in stages:

4 stages of bank MRM development

CCAR banks typically fall into the baseline or optimized stage of MRM development. As banks move through the stages, the drivers of their approach to MRM progress from simple compliance to the business of managing model risk, and from a siloed approach to MRM to a truly integrated MRM framework.

“Only the leading banks are beginning to optimize MRM at this point, and there probably isn’t any bank that’s completely optimizing its model risk management yet.” — Ilieva Ageenko, Leader, Grant Thornton Model Risk ManagementThe six essential components of an optimized MRM framework

Because risk management models change over time, an optimized MRM framework should take a standardized life cycle view of modeling risk. The MRM framework should include standards for model development, implementation, use, validation, ongoing monitoring and maintenance. Effective MRM programs should be based on six components.

  1. Model risk governance and controls: Board members should ensure that the level of model risk is within the institution’s risk appetite tolerance and that all aspects of MRM are covered by suitable policies.
  2. Model risk measurement: This measurement should show how model risk increases with greater model complexity, higher uncertainty about inputs and assumptions, broader use, and larger potential impact.
  3. Model development and implementation: The model risk policy should provide clear standards for model development, model documentation and testing processes.
  4. Model validation: All model components, including input, processing and reporting, should be subject to validation. This applies to both models developed in-house and those obtained from or developed by outside vendors or consultants.
  5. Model use and management: Model users must be aware of limitations and avoid using models inappropriately. Model dashboards should track updated model performance and stages of use within the model life cycle.
  6. Model inventory and mapping: Banks should maintain a firmwide inventory of all models with comprehensive information about models in use, models in development and recently retired models. A map should describe the interconnections of various models and applications.

The next step: Best practices for raising your MRM game
Leading banks are implementing a number of proven best practices across several areas to improve their MRM.

MRM best practices

Steps to take now
There are several steps banks should take immediately to move toward effective MRM.

  • Engage board and senior management: MRM is an increasingly important agenda item for both top management and the board, and the board has a clear role to play in managing model risk. It should understand models’ limitations and challenge their assumptions, critically assess the results, and review and question regulatory recommendations. To help the board fulfill its mandate, senior management must provide information in the appropriate format to assess model risk across the firm. The board’s oversight responsibility requires gaining sufficient insights to assess the firm’s overall model risk profile, which might demonstrate the need for periodic education to remain abreast of enhancements to models.
  • Define and measure KPIs: Measuring the performance of a bank’s MRM efforts on an ongoing basis can only be done with a set of consistent, appropriate performance indicators. Leading banks are going a step further, creating dashboards that display the performance indicator information in a way that makes it readily understandable.
  • Continuously monitor regulatory change: It is essential that banks stay abreast of regulatory changes to ensure that their MRM programs remain in compliance. Outside experience can be helpful in keeping track of those changes.
  • Develop a resource strategy: Banks must craft an effective strategy for sourcing MRM assistance. The strategy should focus on managing goals, staffing and understanding how regulatory expectations are constantly evolving. The ultimate goal is to assemble the resources needed to produce an effective and ongoing MRM framework.

Conclusion: Moving MRM beyond compliance
The ultimate goal for any financial institution should be the creation of a distinct risk management culture. It’s simply not an option anymore — it’s a regulatory necessity. For banks committed to taking the appropriate steps in developing that framework, it’s possible to manage model risk as a core business function within the bank, doing so in a sustainable fashion that can both make the compliance task less onerous and costly while providing valuable insights into the bank’s models and the way they function across the enterprise.

Download the PDF.

Ilieva Ageenko
Managing Director
Financial Services Advisory
Leader, Model Risk Management
T +1 704 632 6820

Nigel Smith
National Leader
Financial Services Advisory
T +1 212 542 9920

Jack Katz
National Managing Partner
Global Leader,
Financial Services
T +1 212 542 9660

1 Office of the Comptroller of the Currency, Supervisory Guidance on Model Risk Management, Apr. 4, 2011.