8.1 What a model is, and what model risk is
▼Definition of a model chapter summary
A model is any method or approach that uses statistical, economic, financial or mathematical theories, techniques and assumptions to transform input data into quantitative estimates. The definition still covers approaches whose inputs are partly or wholly qualitative or based on expert judgement, provided the output is quantitative.
A plain spreadsheet used for arithmetic, such as calculating commission payments, is not a model. Judgement is needed to decide, but even a non-model application needs careful design and maintenance if important decisions rest on it. The IT discipline for this is end-user computing (EUC): helping non-IT staff manage their own-built applications for scalability and robustness.
Models are used beyond risk measurement: informing business decisions (strategy analysis, credit exposure for lending), stress testing (for example capital adequacy), valuing exposures, instruments or positions, and giving real-time signals for algorithmic trading. Machine-learning models are growing in banking and finance, which has made model risk management a top priority.
Model risk and its two causes chapter summary
Model risk is the possibility of adverse consequences arising from decisions based on models that are incorrect or misused. It can cause financial loss, poor business and strategic decisions, or reputational damage. The workbook classes it as an operational risk, arising mainly in two ways:
| Cause | What goes wrong |
|---|---|
| 1. Fundamental errors | The model is inaccurate against its design objective. Errors can enter anywhere from design (theory, sample design, numerical routines, choice of inputs and estimation) to implementation in IT systems. Shortcuts and approximations can compromise reliability, and poor input data or wrong assumptions give poor outputs. |
| 2. Incorrect or inappropriate use | A model that is sound and accurate for its design purpose still carries high model risk if it is misapplied. |
Model risk is managed like other risks: identify the sources, assess the magnitude, then manage it (the governance methods are in section 5 of this page).
8.2 Benefits and limitations of modelling (LTCM, AI)
▼The benefits of modelling syllabus 8.1.1
The world is too complex for the mind to handle whole, so a model gives a simplified representation of the important features, as a map represents terrain. Benefits: a proprietary model that clarifies a complicated reality can be a competitive advantage; it speeds decision-making by automating complex methods; it improves accuracy and repeatability; and repeatability enables "what-if" analysis by varying individual inputs.
AI benefits (workbook): helping firms optimise operations, shed manual processes and move faster, solving difficult problems with minimal human input, building automated and intelligent systems, improving customer experience, and learning from interactions to improve performance.
LTCM: advantage turned into failure syllabus 8.1.1
Long Term Capital Management is the workbook's example of both sides of modelling.
| Phase | What happened |
|---|---|
| Success | A model-based hedge fund with very high early returns; most partners held PhDs; board members included Myron Scholes and Robert Merton (Nobel prize for derivative valuation). It used complex models for fixed-income arbitrage "convergence trades", usually in government bonds. Major banks were investors. |
| Drift | Its capital grew fast but there were only limited convergence trades, so it moved into more aggressive strategies unlike those that built its reputation. By late 1997 it held very large positions in areas such as merger arbitrage. |
| Collapse | Research suggests it lost $4.4bn of $4.7bn capital in under four months in 1998, following the Russian financial crisis; the fund closed in early 2000. Investor Merrill Lynch warned that mathematical risk models may give more security than warranted. |
The lesson: misuse or over-reliance on a model, especially using output for purposes it was not designed for, can produce highly sub-optimal outcomes.
Limits and hidden assumptions syllabus 8.1.1
Models simplify reality, so outputs must not be reused for other purposes. Users must understand the limiting assumptions:
| Assumption | Why it matters |
|---|---|
| Shape of the distribution | If returns are assumed normal, the model suits badly instruments whose returns are not normal. |
| Past predicts future | Many models assume the future resembles the past, and this may be hidden from end users. |
| Business environment at design time | Unless updated, usefulness fades. Example 1: a VaR model coded with a fixed spread when traders took little basis risk misses it once traders take significant basis risk. Example 2: a broker expanding derivatives into emerging markets that keeps old pricing models underestimates hedging cost because of low liquidity. |
Confidence-level limitation: a 95% loss figure of $5m says nothing about how much could be lost on the 5 days in 100 when it is exceeded, only that it will be more than $5m. That is why stress testing is vital for extreme outcomes.
Feedback speed: the shorter the gap between acting on a model and being able to judge the result, the easier to gauge accuracy. In high-frequency trading, P&L is calculated in real time, so models can be shut down automatically if P&L dips below a set level.
AI limitations (workbook): possible bias leading to inaccurate decisions, need for large training datasets, complexity and poor interpretability (hard to understand or spot errors), and high development and maintenance cost.
8.3 Operational risk scenario modelling and Monte Carlo
▼Operational risk scenario model syllabus 8.1.2
Firms must hold capital against unexpected operational losses (see Chapter 2); scenario modelling is one way to size it. Risk workshops with senior risk and business staff consider plausible but unlikely scenarios and record, for each: the frequency, the typical loss, and the severity of an extreme case.
Key points:
- Estimates are then stressed, converting each scenario's impact into a non-normal distribution, because the real impact could exceed the extreme estimate and the event could happen more than once in a year.
- Operational loss events are not normally distributed; they have fat tails (many small losses, few catastrophic). Scenario impacts are often simulated with a lognormal distribution.
- Workshop participants can draw on only about 20 years of experience, so their extreme estimate is a 1-in-20-year (95% confidence) view.
- Firms may need capital for a higher severity: 1 in 200 (99.5%) or 1 in 1,000 (99.9%) are common. Estimates are "stretched" in a controlled, repeatable way using Monte Carlo simulation.
How the Monte Carlo process works syllabus 8.1.2
The workbook uses 100,000 iterations (each iteration a simulated "year") to keep the maths simple; in practice more are needed so neighbouring values at the chosen confidence level are close enough to be stable. Frequency and impact are handled separately:
| Question | Tool | How |
|---|---|---|
| Does the scenario occur this year? (likelihood) | Poisson distribution | Over all iterations the scenario occurs at its estimated frequency (for example 10%), but with realistic clustering: it may occur twice in a decade, then not for 30 years. Needed so several scenarios can hit in the same year. |
| How large is it if it occurs? (impact) | Lognormal curve | The typical estimate is the mode; the extreme estimate fixes a point at the 95% confidence level. A random impact is drawn each time the scenario occurs. |
- 1 99.5% of 100,000 = 99,500, so take the 99,500th row from the bottom (equivalently the 500th worst year).
- 2 This is the one-in-200-year loss from all scenarios together over a one-year horizon. The scenario that drives that row need not be at its own 1-in-200 worst.
- 3 To get each scenario's own figure, sort that scenario's impacts separately and pick its 99,500th row. Adding the individual 99.5% figures gives a bigger total than the combined 99.5% figure, unless correlation is 1. The gap is the portfolio effect (diversification benefit), because not all scenarios happen in the same year.
8.4 Credit VaR, market VaR and Liquidity at Risk
▼Credit VaR syllabus 8.1.2
Example: predict a bond's value in one year from its probability of moving between credit ratings. Rating agencies publish the credit migration probabilities; knowing the possible ratings gives the corresponding bond values. Because migration is only a probability, a published standard deviation lets a distribution be plotted and the value read off at the required probability. For a bond portfolio, correlations between bonds are also needed to capture any diversification benefit. A typical statement: "credit VaR over one year is $5m at the 95% confidence level". Confidence levels are expressed as the percentage chance that a value will not be exceeded.
Market VaR and historical simulation syllabus 8.1.2
Market VaR expresses the maximum market loss that can occur with a specified confidence over a specified period. Example: a one-day 5% VaR of $1m means the probability the portfolio loses more than $1m in a day is 0.05.
Historical simulation looks back at what actually happened. Steps:
- Identify the risk factors (individual stock prices, volatilities, correlations).
- Select a sample of actual historical changes, for example the last 500 trading days.
- Apply each day's change to today's risk factor values, revaluing the current portfolio once per day in the sample.
- Order the resulting portfolio values and, for 95% confidence, read the fifth percentile in the left tail.
- 1 5% of 500 = 25 outcomes. The VaR return is the level at which about 25 outcomes are that bad or worse.
- 2 21 outcomes are −24% or worse (too few) and 43 are −18% or worse (too many), so the cut-off lies between −24% and −18%, nearer the −24% end.
- 3 Read it as: with 95% confidence, the loss will not exceed that level on any one period; on the remaining 5% of periods it will be exceeded.
Liquidity at Risk (LaR) syllabus 8.1.2
The same VaR idea, applied to funding shortfalls, is used because a maturity ladder (Chapter 7) cannot predict uncertain cash flows. Define brackets of funding amounts, count how often actual net funding over a set interval fell in each bracket (for example $20m to $16m counted 14 times, $16m to $12m counted 18 times), and arrange the counts as a distribution with a cluster in the middle and tails either side. As with market and credit VaR, it shows the likely funding requirement at different confidence levels.
| Model | Estimates | Key input |
|---|---|---|
| Operational scenario model | Capital for unexpected operational loss | Workshop estimates of frequency, typical and extreme loss; Poisson + lognormal + Monte Carlo |
| Credit VaR | Portfolio value / loss from rating moves | Rating-migration probabilities, standard deviation, correlations |
| Market VaR | Maximum market loss at a confidence and horizon | Risk factors; for historical simulation, e.g. 500 days of changes |
| Liquidity at Risk | Likely funding requirement | Historic funding amounts placed in brackets |
8.5 Governance of risk modelling
▼The governance framework syllabus 8.1.3
The workbook draws on the US Federal Reserve's Supervisory Guidance on Model Risk Management. A sound framework covers (1) roles and responsibilities of the board, senior management, internal audit and external resources, and (2) policies and procedures, with a model inventory and good documentation. Strong governance is fundamental: even with satisfactory development, implementation, use and validation, weak governance reduces overall effectiveness.
Who does what syllabus 8.1.3
| Party | Responsibility |
|---|---|
| Board of directors | Ultimately responsible; sets the firm-wide framework including standards for development, implementation, use and validation; ensures model risk is within risk appetite and directs changes; sets the tone. Generally delegates execution and maintenance to senior management. |
| Senior management | Establish policies and ensure adherence; assign competent staff with clear model ownership and manage conflicts of interest; oversee development and implementation; set model risk controls (evaluate results, ensure effective challenge, review validation and audit findings, act promptly); consider provisions for trades where the model is known to have limited capability to represent reality. |
| Internal audit | Assesses the overall effectiveness of the framework, for individual models and in aggregate; staff need adequate modelling expertise. Staff who do day-to-day validation should not assess the overall framework (independence). |
| External resources | May help execute activities, but risk management remains internal: agree a written scope of work, have a designated internal party able to understand and evaluate external results, and keep a contingency plan if the resource becomes unavailable or unsatisfactory. |
Policies, procedures and documentation syllabus 8.1.3
Board-level policies should cover all aspects: definition of model risk, assessment of model risk, acceptable practice for development, implementation and use, appropriate validation activities, and governance and controls over the process.
- Emphasise testing and analysis: targets for model accuracy, standards for acceptable discrepancy, and procedures to review and respond to unacceptable discrepancies.
- Set the priority, scope and frequency of validation: how much before models go into production, and the scope of ongoing validation.
- Back-testing for models based on history: benchmark outputs against reality once reality has caught up. A model giving VaR for one week ahead is compared with the actual market position a week later.
- Set validation requirements for vendor and third-party models.
- Require detailed documentation: a model inventory, results of modelling and validation, and issues and their resolution.
- The board or its delegates periodically review the policies for consistent, rigorous practice.
8.6 Ch 8 cheat sheet
▼All the definitions + numbers
| Item | Answer |
|---|---|
| Model | Method turning input data into quantitative estimates (qualitative inputs allowed if output is quantitative) |
| Spreadsheet arithmetic | Not a model (still needs EUC care) |
| Model risk | Adverse consequences from decisions based on incorrect or misused models |
| Model risk type | Operational risk |
| Two causes | Fundamental errors; incorrect or inappropriate use |
| Benefits | Simplified view, competitive edge, speed, accuracy, repeatability, what-if |
| LTCM | Convergence trades, then drift to merger arbitrage; lost $4.4bn of $4.7bn in 1998; closed early 2000 |
| Limiting assumptions | Distribution shape; past predicts future; business environment at design |
| Confidence-level blind spot | VaR says nothing about losses beyond the cut-off; use stress testing |
| AI limitations | Bias, data needs, opacity, cost |
| Op risk scenario inputs | Frequency, typical loss, extreme-case severity |
| Op risk distribution | Fat-tailed, not normal; lognormal often used |
| Workshop extreme estimate | ~20 years experience = 1 in 20 = 95% |
| Capital severity levels | 1 in 200 = 99.5%; 1 in 1,000 = 99.9% |
| Poisson vs lognormal | Poisson = frequency; lognormal = impact |
| 99.5% of 100,000 | 99,500th row from the bottom |
| Portfolio effect | Sum of individual 99.5% figures exceeds combined 99.5% figure |
| Credit VaR input | Rating-migration probabilities, standard deviation, correlations |
| Market VaR meaning | One-day 5% VaR $1m: 0.05 probability of losing more than $1m |
| Historical simulation | Risk factors, sample (e.g. 500 days), revalue, read 5th percentile for 95% |
| Governance guidance | US Federal Reserve Supervisory Guidance on Model Risk Management |
| Board vs senior management | Board owns and sets appetite; senior management executes and maintains |
| Internal audit | Assesses framework; validators should not assess the overall framework |
| Back-testing | Compare model output to reality once it has caught up |