Models and their limits

We need models, but let’s not mistake them for truth.

Models are an essential tool in banking and insurance. However, it is important that we always keep in mind that any model that we use can only be an approximation to a much more complex reality. The statistician George Box once famously remarked: “All models are wrong but some are useful”1. This was stated many decades ago and still holds true today. It tells us two things.

First, do not throw in the modelling towel simply because you can never know the “true” model (if there really is such a thing). Instead, a well-crafted approximation can be very useful in the work that we do to deliver robust results that can help decision-making. Second, and conversely, reckless use of a badly chosen model can lead to disastrous consequences.

The latter point rings true, with many real-life case studies that litter the history of banking and insurance from the recent failure of banks to model the risks of structured products in the financial crisis, to Equitable Life and the failure of Long Term Capital Management, the hedge fund that almost single-handedly introduced modern finance to risk modelling and the use of Value at Risk (VaR).

A definition of model risk

The extensive use of modelling in finance means that we must consider the concept of model risk. But we need to have a working definition of what we mean by this expression. The definition I like, and which captures the concept most effectively, is as follows: Model Risk is the risk of financial, reputational or other losses as a result of decisions based on the use of an incorrect model, having failed to acknowledge alternative models, or based on inappropriate use of a model.

Image: photologic / Flickr
Using the broad definition above we consider that model risk has two parts: statistical aspects of model risk, and operational risks. Breaking down model risk to these and even smaller components can then help us explore ways of mitigating model risk.

Statistical model risk

Statistical model risk builds on George Box’s quote above. In this context, we assume that each model we consider is correctly specified and calibrated, and that we can make judgements on which models best describe the past and present states of the world.

Model risk then arises because we focus on a single “best” model, and ignore alternatives. In addition, the randomness inherent in our data can lead us inadvertently to both the wrong model and the wrong calibration of the model.

Operational model risk

In contrast to statistical model risk, operational model risk is derived from a range of errors that occur either during the design of the model or through its use; all forms of operational risk.

Risks associated with the model could include instances where the mathematics underpinning the model is flawed, leading to the use of an incorrect formula or implementation errors. For example, the model is correct but it is incorrectly programmed. It could also result from miscalibrated model parameters (e.g. there is an error in the parameter estimation program).

Operational model risks may also rise from the model user. If users are overconfident in the choice of model and/or its calibration this can lead to excessive leverage in some financial situations. Similarly if the model validation and governance processes are flawed the other operational risks discussed here could flourish.

There are a range of other risks that may develop in the use of the model, including inappropriate use, wrong model inputs or misunderstanding or underestimating a model’s limitations.

The latter point could lead to one of the more common risks: using the model in situations for which the model was not designed. A simple model might be good and robust for a simple application, but it should not be used uncritically for a more complex application. Handling this needs well-crafted constraints on model usage and for these constraints to be effectively communicated.

Model risk mitigation

In order to guard against the risks outlined above we can draw on a number of basic principles to help mitigate model risk.

First and foremost of these is the need to acknowledge other valid models. Reducing the statistical aspects of model risk then means attaching weight to more than one model and allowing for parameter estimation uncertainty in decision-making. Additionally, model risk can be reduced by testing for robustness of your decisions relative to the choice of model and calibration of that model.

What’s more, acknowledgement of the statistical aspects of model risk is likely to dampen down enthusiasm for excessive leverage.

On the operational side a key part of the process is to check if the model passes the use test. That is, is it being used in day-to-day decision-making. Underpinning this is the idea that decision makers are smart guys and have an understanding of model risk. If they thought that model risk had not been assessed adequately, then they would not be using the model to make decisions. On its own, this is not a failsafe criterion, but it is reasonable to see active “use” as being correlated with good model risk management.

Putting in place strong model validation procedures and strong model governance framework including a rigorous regulatory approval processes is also an essential component of the process and should go hand-in-hand with a strong programme of education and communication.

Conclusion

Models are an essential tool for making sense of the world around us, but they should not be confused with the world itself. As such, they are limited tools. But knowing their limits is what makes them so powerful for the user.

Andrew Cairns is Professor of Financial Mathematics at Heriot-Watt University, Edinburgh, and Director of the Actuarial Research Centre. The views expressed in this article are the author’s own.