Models bring benefits, but how do we ensure they’re fit for purpose?
How banks use models
Banks make significant use of models as a key part of their business practice and in various aspects of decision-making:
– Models are widely used to value assets and liabilities, in particular derivative products and loan provisions. They are therefore key to financial reporting.
– Models are integral to many of the risk quantification frameworks and caronsequently are key elements for determining prudential requirements, including the solvency ratio, Pillar 2 and stress testing.
– Models are also used widely in decision making directly affecting customer outcomes, such as customer acquisition decisions, account fraud / compliance screening and arrears collection strategies.
The use of models brings significant benefits including automation, efficiency, objective decision-making and the ability to understand and oversee our risks. For large banks, the role of models is perhaps even more central to the institution: in computing capital requirements, banks have approval from the regulators for the use of internal, risk sensitive models to capitalize appropriately; the use of such models relative to more standardized, risk insensitive treatments brings significant benefits to the real economy as it allows for more ‘efficient’ use of capital.
What is a model?
Providing a precise and comprehensive definition of a model as used in banking is nontrivial given the wide range of contexts that firms apply analytics; indeed this has been the subject of significant debate over the past few years. It does seem now that the industry consensus is to adopt the regulatory definition given by the Board of Governors of the Federal Reserve System and the Office of the Comptroller of the Currency (OCC) in the USA, and to then apply an element of judgment.
A model could then be defined as follows.
A model is a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques and assumptions to process input data into quantitative estimates. In general, a model consists of three components:
1. An information input component, which collects data for the model. This can involve manual inputs or automated data feeds or interfaces;
2. a processing component, which transforms the data and the assumptions into estimates or values. This includes all model choices and specific mathematical calculations and data treatments; and
3. an output or reporting component, which translates the estimates or values into useful business information sometimes in conjunction with policy rules.
Definition of Model risk
As explained by other authors, model usage exposes a financial institution to model risk. Such risk can result in financial and reputational loss to the bank and its customers. It is therefore important to understand both model risk and how to manage it.
In our view there are in effect three fundamental elements of model risk. Models might be specified incorrectly (i.e., inappropriately designed – failing, for example, to capture the economics of the situation adequately); implemented incorrectly (an error in translating a model specification into a version actually used); or used incorrectly (correctly specified but applied inappropriately).
The importance of models to modern banking has been recognized by the regulatory community. Most significantly, in the USA, guidance on model risk was developed and issued jointly by the OCC and the Fed in April 2011. The guidelines were adopted by the OCC as Bulletin 2011-12 and by the Federal Reserve as FRB SR 11-7 and apply to all US institutions as well as foreign banks operating in the US.
SR 11-7 insists on a comprehensive view on model risk management within financial institutions, moving away from a process whereby model governance comprised validation and approval post model development for a subset of the models used within a firm. Banks are now expected to (Kaufmann, 2015):
– Assess all models across the firm
– Develop a strong model risk management governance framework. This should cover robust model development, sound model validation, and strong governance around managing models and the data used in models
– Manage models through their lifecycle i.e., monitor performance, ensuring that models remain fit for purpose
– Ensure active involvement of senior leadership and Internal Audit
– Develop automated process flows over the model lifecycle. This is to ensure sustainability and transparency of the controls and evidence the compliance activities
– Actively benchmark their existing model governance and model development and validation processes against the guidance
In the US, assessments of these frameworks have been carried out as part of specific regulatory engagements and also as part of the Comprehensive Capital Analysis and Review (CCAR processes): models are integral to measurement and hence capitalisation of risk. However, as part of the CCAR process, US regulators have shown equal interest in evaluating the processes around models used to assess revenue generation. Actions to enhance modelling frameworks have been part of the remediation related to CCAR.
The European Union and the UK do not have an equivalent consolidated set of model risk requirements. Instead, there are various references and standards typically found in specific guidance related to areas such as valuation, and market risk and credit risk management. For example,
– the EU capital requirements rules (CRR / CRD IV) specifically mention model risk in the context of operational risk as a requirement for consideration in the Internal Capital Adequacy Assessment Process (Pillar 2).
– Furthermore, in its Stress Testing Guidance (2014) the Bank of England request specific information on the key items that make up a typical Model Risk Framework, in particular documentation on: the firm’s existing stress-testing policies, methodologies, and overall framework across all risk types including roles, responsibilities, governance arrangements, and coverage of portfolios.
Nevertheless, given the US guidance and how it applies to all US firms, irrespective of size, and many significant non-US institutions (given that they do significant amounts of business in the US) we are seeing a convergence in how firms think about model risk.
Elements of a Model Risk Management Framework
The past few years have seen the banking industry take significant steps to address the challenges of enhancing their model risk management. Looking across the industry, a number of key themes emerge, which include: building a complete model inventory – identifying all models across the organisation; providing appropriate governance – the need for model risk assessment; organisation of model governance; repositioning existing Validation functions and defining and embedding Model Risk Appetite. This list is by no means exhaustive. However, anecdotally, these are some of the areas that have been of significant and active debate across the industry and therefore warrant closer scrutiny in my view
Building a Complete Inventory
Although firms use models for a variety of purposes, they have tended to focus on the governance of models used for the measurement of risk and the computation of (regulatory) capital. Given the systematic importance of these models to the firm, this is not unreasonable. However, it has tended to mean that models underlying customer-related decisions, for example, have not necessarily been subject to significant scrutiny.
The changes discussed above have led to firms making significant efforts to determine the model population across the institution. This has typically been achieved through firm-wide surveys, which seek to identify all the models along with clarifying business ownership (e.g. who develops, owns and uses the model).
Particular models are often used for a number of purposes or in different contexts and it is important to ensure that we record these. As an example, derivative-pricing models may be developed by the desk for valuing positions in ‘normal’ markets, while the same models may be used to generate valuations under market stresses. Depending on the form of the stress, certain assumptions underlying the valuation model (e.g. the availability of particular hedging instruments) may no longer be valid, in which case the model may not be appropriate for stress testing.
Providing appropriate governance – the need for model risk assessment
There are perhaps three fundamental underlying components in model risk: the intrinsic uncertainty introduced in choosing a particular model, risks associated with the design and implementation of the model and the risks in applying the model in practice.
Hence in seeking to manage model risk we need to implicitly understand, quantify and accept the first component, control the design risk by insisting on appropriate development and implementation standards as well as insisting on an appropriately robust software platform, and control the use risk by insisting on appropriate business process and governance.
It is also critical to understand the drivers of model risk associated with each model. Probably the first step is to gather basic information describing the model and its operating environment. We describe some of the information we might be interested in below:
– Materiality – How important is the model – example measures of materiality include: Balance Sheet, Profit & Loss, capital, RWA coverage, exposure, exposure per annum, number of customers.
– Regulatory – Is the model explicitly subject to regulations, what is the level of compliance
– Data – Accuracy and completeness of data used to build and support the on-going use of the model
– Systems and Implementation – What system(s) is the model implemented in? Use of Strategic Infrastructure versus e.g., implementation in Excel.
– Use – List of uses, especially if as an input to other processes/models. Volume and extent of overrides of model outputs.
– Documentation – Does appropriate documentation exist? Qualitatively is there good institutional understanding of the model? Is the model industry standard?
This provides us with a model risk assessment and a means of understanding the state of our model portfolio and where, for example, we may need to expend effort in developing the control environment (be it in terms of enhanced documentation, model testing, stronger implementation practice, more validation work etc.). We can also use this risk assessment to ensure that the amount of extra work / governance that we associate with a particular model / suite of models is proportionate with the risk associated with the model.
Organisation of Model Governance: three lines of defence
The majority of banks have moved towards a more formalized three ‘Lines of Defence’ (LoD) model for the governance of models:
The first LoD consists of the Model Owners, Developers, and Users within the lines of business. Their responsibilities are defining, developing, implementing, and operating the model, monitoring its performance, and managing changes
The second LoD is the Model Risk function, whose responsibilities are:
– establishing policies and standards,
– performing model risk assessment,
– managing an inventory of models,
– independent monitoring of model performance, model usage, and adherence to management policies
– reporting to board/senior management
The role of the third LoD is filled by Internal Audit, whose responsibility is to provide an independent assessment of the adherence of the first and second LoD’s to Model Risk Policies.
Critically it is important to see the process of model governance as more than simply a bilateral conversation between the model developers and model validators. The involvement of model users is critical. They ensure that the model developers have captured (to an agreed extent) the key features of the real world problem that the model attempts to abstract and that this remains the case on an on-going basis. As such governance structures should be designed to facilitate this process as far as is possible.One current area of debate is whether models should be approved by individuals or by committee. For example, as firms look to simplifying decision making and individual accountability there is a push to making individuals responsible. This process does, however, ensure senior stakeholder focus and pushes firms towards understanding their key models.
Challenges for the Validation Function
Rolling out an enhanced model risk framework also places significant demands on model validation groups. Staff carrying out validation should have the requisite knowledge, skills, and expertise and given the complexity of many models this frequently entails a high level of technical expertise. These staff should also have a significant degree of familiarity with the line of business using the model and the model’s intended uses. Given the significant expansion in remit, particularly in terms of different businesses and new uses previously not considered, the challenge then is of challenging validation functions to understand a significantly larger proportion of the business to a greater extent, and to understand precisely how models inform business decision making rather than a pure technical challenge. This necessarily entails interaction with a larger number of stakeholders, many of whom will not necessarily be familiar with model risk disciplines.
Model Risk Appetite
To manage Model Risk in a manner that is identical to other key risk types, such as market risk or credit risk, we need to establish a “Model Risk Appetite Framework”, just as we do for those other risks.
The Financial Stability Board (FSB) has listed the key component of a risk appetite framework (Principles for an Effective Risk Appetite Framework, November 2013), which are equally applicable to model risk appetite. The key elements are establishing a:
– Risk Appetite Statement – a description of the overall approach whereby risk appetite is established, communicated and monitored. It requires us to elucidate how we quantify model risk.
– Risk Capacity – the maximum level of risk that can be assumed before breaching capital/liquidity constraints or regulatory/stakeholder obligations
– Risk Appetite – a statement of the levels of risk the firm is willing to assume within the risk capacity in order to achieve strategic objectives
– Risk Limits – allocating the risk appetite to specific areas at lower levels
– Risk Profile – a view of the firm’s current risk exposures. This would need to consider the impact of netting, mitigants and should incorporate appropriate aggregation.
A plausible approach to quantification relies on the information framework discussed above; clearly one needs to use some form of scoring to represent qualitative information such as the level of documentation. We should also try to gauge the levels of conservatism built into these models (i.e., consider the economic risk to th e bank) to ensure a consistent comparison across models. We note that such frameworks are as yet emergent.
Furthermore, it is critical to understand how well our framework of models perform under ‘stress’; to do this we need to understand both the underlying key economic assumptions of any given model, but also to understand how the outputs of these models feed into other models.
As model use becomes more established there is a growing trend towards standardization of practices for their regulation and use. Over time the concept of managing model risk is becoming more established, building on the existing risk management strategies. Model use and governance is an evolving practice that is likely to continue as they become ever more entrenched in banking and finance in general.
Dherminder Kainth is head of the Quantitative Research Centre (QuaRC) at the Royal Bank of Scotland Group.
The opinions and views expressed herein are those of the author and do not necessarily represent those of RBS.