Who will mind the models?

 

Strategies for Deploying Model Risk Management (MRM)

 

Financial professionals are increasingly using models for both financial and operational analysis. Models help them manage the risks of capital allocations and underwriting. They facilitate compliance with regulatory reporting requirements. Unfortunately, like all analytical tools, they pose risks of their own.

Not surprisingly, the Federal Reserve and Office of the Comptroller of the Currency (OCC) weighed in, in the form of supervisory guidance letter SR 11-7. The letter describes a model as “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information.”

In practice, models can be thought of as quantitative simplifications of real-world mechanics and potential outcomes. Of course, simplification can lead to distortion, and those distortions can subvert the very purpose of models: to accurately predict outcomes.

That is precisely where Model Risk Management (MRM) can help.

Defined as the analysis and mitigation of risk associated with models used to make decisions, MRM optimizes model outcomes and reduces confidence error. But MRM also promotes institutions to generate effective governance structures, policies, and procedures; conduct data quality and model performance reviews; and establish ongoing monitoring and reporting standards. By using sound MRM practices, decision makers can fully understand the limitations of the models they use and contextualize the performance of these models over time.

Ultimately, this should lead to increasingly well-informed decisions through detailed validations of models that test the model’s assumptions and the completeness and accuracy of its results. And that can translate to fine-tuned loan underwriting decisions and improved identification of possible money laundering operations.

Since models have become more prevalent, MRM is imperative to the stability of our financial system. The supervisory guidance letter SR 11-7 provides recommendations for managing model risk in the following areas:

  1. Governance
  2. Data
  3. Model Performance
  4. Conceptual soundness
  5. Ongoing monitoring and reporting

 

 

Federal regulators have broadly interpreted what is considered a model for purposes of SR 11-7.

Governance

 

Model governance is foundational to effective model risk management practices. Good governance structures can facilitate compliance with current laws and regulations through accountability, documentation standards, and oversight. Specifically, the modeling process should have model owners, institutional accountability (including for vendor models), an up-to-date model inventory, stakeholder involvement, appropriate management approval, and contingency plans.

Federal regulators have broadly interpreted what is considered a model for purposes of SR 11-7. Based on the regulators’ definition, any “quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates” is a model.

The definition goes further to encompass many internal processes that some organizations would refer to as a “tool” rather than a model. In SR 11-7 regulators further clarify that, “the definition of model also covers quantitative approaches whose inputs are partially or wholly qualitative or based on expert judgement, provided the output is quantitative in nature.”

For example, some institutions would consider a Customer Risk Rating methodology for BSA/AML purposes a “tool” because “traditional” modeling techniques (e.g., regression) are not being used to interpret the data. However, under SR 11-7 guidance, this type of analytical tool should be considered a “model.” The Credit Risk Rating methodology is considered a “model” because the analysis has elements that are subject to model risk: 1) an information input component (qualities of the customer); 2) a processing component (translating those properties into a score); and 3) a reporting component (categorizing low, medium, and high-risk customers). There is model risk any time varying data points are used to create a quantitative measure (a risk score) to make a qualitative judgement (riskiness of a customer).

After determining which methods, systems, and approaches are eligible for MRM, the organization must systemize the management and oversight of these functions through appropriate documentation and accountability.

 

 

 

There is model risk any time varying data points are used to create a quantitative measure (a risk score) to make a qualitative judgement (riskiness of a customer).

1 Policies and Procedures

 

Given this level of regulatory attention, it’s important to formalize policies and procedures that govern the modeling process. The goal is to generate models which consistently produce reliable results. When designing policy and procedure coverage, financial services institutions should ensure these topics are covered:

 

  • Roles and Responsibilities/Approvals: Indicate “Who approves a model for use?” and “Who approves changes to the model?”
  • Model Inventory: Maintain a centralized model inventory that includes all models used, the generic risk ratings, and other relevant information (e.g., last validation date, next scheduled validation date, etc.).
  • Documentation: Establish guidelines for model development documentation, what elements the documentation should contain, and the person or team responsible for approving the documentation for use.
  • Risk appetite: Specify the acceptable level of risk. Typically, this will also outline the model risk rating process.
  • Model validation procedures: Determine how often models of each risk rating are validated, which department is responsible for working with independent validators, etc.
  • Contingencies: Identify who escalates the situation and decides what to do if a model is not working properly, what action is taken if policies and procedures are not followed (e.g., if a model is not validated on time, who is notified to ensure immediate action is taken to rectify), and who becomes the model owner if key personnel exit the organization?


These principles are not the only areas where financial institutions need to create policies and procedures. Other elements may be jurisdiction dependent, operationally specific, or reflective of the specific type of institution.

 

 

 

Banks often rely on boiler-plate model manuals from the vendor that provide insufficient detail on the model’s methodology, limitations, and controls.

2 Documentation

 

Institutions also frequently struggle with model documentation. Especially in the case of a vendor-supplied model, banks often rely on boiler-plate model manuals or user procedures from the vendor that provide insufficient detail on the model’s methodology, limitations, and controls. The goal of model documentation is to communicate the function and methodology of a model to both users and non-users alike. These best practices will help achieve that goal:

  • Model Owner: Who has ultimate accountability for model use and performance?
  • Model Purpose: What is the model used for and, equally, what is it not used for? For example, a Fair Lending Model should be used to determine disparate impacts in mortgage lending but should categorically not be used for underwriting purposes.
  • Assumptions and Limitations: The assumptions that inform a model are too frequently left undocumented, even though they usefully contextualize inputs and outputs. For example, an underlying assumption of a portfolio projection model might be that the model only holds if no economic shocks occur.
  • Model Methodology: Discuss the methodology selected, alternatives considered, and justification for the method settled upon. You should be able to explain your model to colleagues with a non-quantitative background. For less complex models, this can be as simple as documenting that the methodology is standard practice, and no serious alternatives exist.
  • Data: Document the sources of data used (internal and external), any transformations performed, and any limitations or assumptions in the data.
  • Final model and outputs: Document the model that was ultimately created, and define all inputs, all outputs, and any relevant details for operation. An uninvolved third party should be able to look at this section and, with the relevant data, completely reproduce the operation of the model independently.
  • Model testing – Define any model performance/assumption testing being performed. Then, report on and interpret the results.
  • Model Change Log – When was the documentation last updated? What was changed from the previous version? Who approved the change?


Since these documents inform senior leadership, external parties, and regulators, consistent and thorough documentation practices are imperative.

 

 

You should be able to explain your model to colleagues with a non-quantitative background.

3 Change Management

 

Change happens—often in the form of regulatory updates and emerging industry trends, sometimes in the form of a worldwide pandemic. Institutions should position themselves to address these changes through model upgrades or redevelopment. Both large and small updates can present new risk exposures, including third-party and operational risk. Fortunately, the change management process provides a structured way to implement model-related changes and minimize those risk.

An oversight process can ensure key stakeholders agree with — and authorize — the model change while considering the priorities of those affected (such as parties external to the department who use the model, Senior Management Committees, or the Board of Directors.) Consider engaging a third-party vendor to support the process.

Third-party risk can be a special problem for models developed by vendors, housed within their platform, subject to their policies and procedures, and then updated by the vendor—especially if your institution has restricted access to the underlying logic. To mitigate this risk, require the vendor to formally communicate model changes prior to implementation. Also, periodically request results from independent reviews of vendor models, including SOC reports or audit reports in which change controls are included in the scope.

These measures help ensure that changes to models and their effect on decision making are considered by model users and appropriate decision makers.

 

 

 

4 Oversight and Supervision

 

Day-to-day MRM is often the work of subject matter experts in analytics, valuation, and compliance. However, even the best processes and controls are only effective if they are championed by leadership. Board and management oversight of models is critical. As the OCC states in their MRM handbook, model risk governance should reflect a sound corporate culture.

The board of directors is responsible for setting the tone and overseeing management’s role in fostering and maintaining a sound, appropriately prudent, and ethically scrupulous corporate culture. MRM activities should be a regular topic for the board of directors and relevant management committees, such as risk management or compliance committees. The MRM Policy should be reviewed and approved by the Board and any relevant management committees annually. The board and management do not need to know every detail of every model, but they do need to know the risk presented by the models as a whole. The Board must determine which models have the greatest potential impact on company financials and reputation.

Model validations and annual reviews will help them assess the overall risk presented, the business processes the models support, and resources needed to solve problems or explore opportunities for improvement. When model risk management is a set agenda topic, the deliberations by the board and management will be recorded in the minutes. This will provide written evidence of their oversight and awareness of model risk for future reference and reviews with regulators.

 

 

 

Conclusion

 

Effective model risk management starts with governance. The design of policies, procedures, and model documentation create consistent output from models that help institution leadership make better and more informed decisions. Effective change management and oversight allows for a transparent model creation and development process, which allows leadership to be more in tune with the risks the business is engaged in. These are the first steps that must be taken by any institution to implement model risk management for more informed decision making throughout the firm. As models continue to become more integral to the operations of instructions so does the importance of these elements for good governance.

 
 

Contacts:

 
 
 
 
 

Our featured risk, compliance and controls insights