Chapter 8 · Model Risk

4 of 100 exam questions (~4%) · what a model is · benefits and limits · LTCM · operational risk scenario model and Monte Carlo · credit VaR · market VaR · Liquidity at Risk · model governance
← Back to quiz
Why this chapter matters. The lightest chapter: 4 of 100 questions (~4%) in the v12 weighting used by this course (the Edition 11 workbook itself states about 3). Because it is short, a few clean definitions win marks. Trap zones: model risk is an operational risk, it has two causes: a flawed model, or a sound model misused, a simple spreadsheet calculation is not a "model", operational risk losses are fat-tailed (lognormal used), not normal, Poisson governs how often a scenario occurs, lognormal governs how bad it is, historical simulation VaR reads the 5th percentile for 95% confidence, and the board owns the framework but delegates execution to senior management. Cross-links: VaR and Monte Carlo come from Chapter 5, credit ideas from Chapter 4, LaR from Chapter 7.

8.1 What a model is, and what model risk is

▼

Definition of a model chapter summary

A model is any method or approach that uses statistical, economic, financial or mathematical theories, techniques and assumptions to transform input data into quantitative estimates. The definition still covers approaches whose inputs are partly or wholly qualitative or based on expert judgement, provided the output is quantitative.

A plain spreadsheet used for arithmetic, such as calculating commission payments, is not a model. Judgement is needed to decide, but even a non-model application needs careful design and maintenance if important decisions rest on it. The IT discipline for this is end-user computing (EUC): helping non-IT staff manage their own-built applications for scalability and robustness.

Models are used beyond risk measurement: informing business decisions (strategy analysis, credit exposure for lending), stress testing (for example capital adequacy), valuing exposures, instruments or positions, and giving real-time signals for algorithmic trading. Machine-learning models are growing in banking and finance, which has made model risk management a top priority.

Model risk and its two causes chapter summary

Model risk is the possibility of adverse consequences arising from decisions based on models that are incorrect or misused. It can cause financial loss, poor business and strategic decisions, or reputational damage. The workbook classes it as an operational risk, arising mainly in two ways:

CauseWhat goes wrong
1. Fundamental errorsThe model is inaccurate against its design objective. Errors can enter anywhere from design (theory, sample design, numerical routines, choice of inputs and estimation) to implementation in IT systems. Shortcuts and approximations can compromise reliability, and poor input data or wrong assumptions give poor outputs.
2. Incorrect or inappropriate useA model that is sound and accurate for its design purpose still carries high model risk if it is misapplied.

Model risk is managed like other risks: identify the sources, assess the magnitude, then manage it (the governance methods are in section 5 of this page).

Memory hook: wrong model, or wrong use. A correct model used for the wrong job is still model risk.
Quick check
Which statement about model risk is consistent with the workbook?

8.2 Benefits and limitations of modelling (LTCM, AI)

▼

The benefits of modelling syllabus 8.1.1

The world is too complex for the mind to handle whole, so a model gives a simplified representation of the important features, as a map represents terrain. Benefits: a proprietary model that clarifies a complicated reality can be a competitive advantage; it speeds decision-making by automating complex methods; it improves accuracy and repeatability; and repeatability enables "what-if" analysis by varying individual inputs.

AI benefits (workbook): helping firms optimise operations, shed manual processes and move faster, solving difficult problems with minimal human input, building automated and intelligent systems, improving customer experience, and learning from interactions to improve performance.

LTCM: advantage turned into failure syllabus 8.1.1

Long Term Capital Management is the workbook's example of both sides of modelling.

PhaseWhat happened
SuccessA model-based hedge fund with very high early returns; most partners held PhDs; board members included Myron Scholes and Robert Merton (Nobel prize for derivative valuation). It used complex models for fixed-income arbitrage "convergence trades", usually in government bonds. Major banks were investors.
DriftIts capital grew fast but there were only limited convergence trades, so it moved into more aggressive strategies unlike those that built its reputation. By late 1997 it held very large positions in areas such as merger arbitrage.
CollapseResearch suggests it lost $4.4bn of $4.7bn capital in under four months in 1998, following the Russian financial crisis; the fund closed in early 2000. Investor Merrill Lynch warned that mathematical risk models may give more security than warranted.

The lesson: misuse or over-reliance on a model, especially using output for purposes it was not designed for, can produce highly sub-optimal outcomes.

Quick check
What is the main model-risk lesson of LTCM in the workbook?

Limits and hidden assumptions syllabus 8.1.1

Models simplify reality, so outputs must not be reused for other purposes. Users must understand the limiting assumptions:

AssumptionWhy it matters
Shape of the distributionIf returns are assumed normal, the model suits badly instruments whose returns are not normal.
Past predicts futureMany models assume the future resembles the past, and this may be hidden from end users.
Business environment at design timeUnless updated, usefulness fades. Example 1: a VaR model coded with a fixed spread when traders took little basis risk misses it once traders take significant basis risk. Example 2: a broker expanding derivatives into emerging markets that keeps old pricing models underestimates hedging cost because of low liquidity.

Confidence-level limitation: a 95% loss figure of $5m says nothing about how much could be lost on the 5 days in 100 when it is exceeded, only that it will be more than $5m. That is why stress testing is vital for extreme outcomes.

Feedback speed: the shorter the gap between acting on a model and being able to judge the result, the easier to gauge accuracy. In high-frequency trading, P&L is calculated in real time, so models can be shut down automatically if P&L dips below a set level.

AI limitations (workbook): possible bias leading to inaccurate decisions, need for large training datasets, complexity and poor interpretability (hard to understand or spot errors), and high development and maintenance cost.

8.3 Operational risk scenario modelling and Monte Carlo

▼

Operational risk scenario model syllabus 8.1.2

Firms must hold capital against unexpected operational losses (see Chapter 2); scenario modelling is one way to size it. Risk workshops with senior risk and business staff consider plausible but unlikely scenarios and record, for each: the frequency, the typical loss, and the severity of an extreme case.

Key points:

  • Estimates are then stressed, converting each scenario's impact into a non-normal distribution, because the real impact could exceed the extreme estimate and the event could happen more than once in a year.
  • Operational loss events are not normally distributed; they have fat tails (many small losses, few catastrophic). Scenario impacts are often simulated with a lognormal distribution.
  • Workshop participants can draw on only about 20 years of experience, so their extreme estimate is a 1-in-20-year (95% confidence) view.
  • Firms may need capital for a higher severity: 1 in 200 (99.5%) or 1 in 1,000 (99.9%) are common. Estimates are "stretched" in a controlled, repeatable way using Monte Carlo simulation.
Trap: participants' extreme estimate is 95% (1 in 20), not 99.5%. Monte Carlo stretches it out to the required level.

How the Monte Carlo process works syllabus 8.1.2

The workbook uses 100,000 iterations (each iteration a simulated "year") to keep the maths simple; in practice more are needed so neighbouring values at the chosen confidence level are close enough to be stable. Frequency and impact are handled separately:

QuestionToolHow
Does the scenario occur this year? (likelihood)Poisson distributionOver all iterations the scenario occurs at its estimated frequency (for example 10%), but with realistic clustering: it may occur twice in a decade, then not for 30 years. Needed so several scenarios can hit in the same year.
How large is it if it occurs? (impact)Lognormal curveThe typical estimate is the mode; the extreme estimate fixes a point at the 95% confidence level. A random impact is drawn each time the scenario occurs.
Worked example: reading the 1-in-200 loss
After 100,000 iterations, the total impact of all scenarios that occurred in each iteration is summed and sorted with the worst at the top. Which row gives the 99.5% (1-in-200) figure, and what does adding up each scenario's own 99.5% figure show?
  1. 1 99.5% of 100,000 = 99,500, so take the 99,500th row from the bottom (equivalently the 500th worst year).
  2. 2 This is the one-in-200-year loss from all scenarios together over a one-year horizon. The scenario that drives that row need not be at its own 1-in-200 worst.
  3. 3 To get each scenario's own figure, sort that scenario's impacts separately and pick its 99,500th row. Adding the individual 99.5% figures gives a bigger total than the combined 99.5% figure, unless correlation is 1. The gap is the portfolio effect (diversification benefit), because not all scenarios happen in the same year.
Quick check
In the operational risk Monte Carlo example, the Poisson distribution is used to model:

8.4 Credit VaR, market VaR and Liquidity at Risk

▼

Credit VaR syllabus 8.1.2

Example: predict a bond's value in one year from its probability of moving between credit ratings. Rating agencies publish the credit migration probabilities; knowing the possible ratings gives the corresponding bond values. Because migration is only a probability, a published standard deviation lets a distribution be plotted and the value read off at the required probability. For a bond portfolio, correlations between bonds are also needed to capture any diversification benefit. A typical statement: "credit VaR over one year is $5m at the 95% confidence level". Confidence levels are expressed as the percentage chance that a value will not be exceeded.

Market VaR and historical simulation syllabus 8.1.2

Market VaR expresses the maximum market loss that can occur with a specified confidence over a specified period. Example: a one-day 5% VaR of $1m means the probability the portfolio loses more than $1m in a day is 0.05.

Historical simulation looks back at what actually happened. Steps:

  1. Identify the risk factors (individual stock prices, volatilities, correlations).
  2. Select a sample of actual historical changes, for example the last 500 trading days.
  3. Apply each day's change to today's risk factor values, revaluing the current portfolio once per day in the sample.
  4. Order the resulting portfolio values and, for 95% confidence, read the fifth percentile in the left tail.
Worked example: the 5th percentile of 500 outcomes
With 500 simulated outcomes, the workbook's histogram shows 43 returns of −18% or worse and 21 of −24% or worse. Roughly where does the 95% VaR sit?
  1. 1 5% of 500 = 25 outcomes. The VaR return is the level at which about 25 outcomes are that bad or worse.
  2. 2 21 outcomes are −24% or worse (too few) and 43 are −18% or worse (too many), so the cut-off lies between −24% and −18%, nearer the −24% end.
  3. 3 Read it as: with 95% confidence, the loss will not exceed that level on any one period; on the remaining 5% of periods it will be exceeded.

Liquidity at Risk (LaR) syllabus 8.1.2

The same VaR idea, applied to funding shortfalls, is used because a maturity ladder (Chapter 7) cannot predict uncertain cash flows. Define brackets of funding amounts, count how often actual net funding over a set interval fell in each bracket (for example $20m to $16m counted 14 times, $16m to $12m counted 18 times), and arrange the counts as a distribution with a cluster in the middle and tails either side. As with market and credit VaR, it shows the likely funding requirement at different confidence levels.

ModelEstimatesKey input
Operational scenario modelCapital for unexpected operational lossWorkshop estimates of frequency, typical and extreme loss; Poisson + lognormal + Monte Carlo
Credit VaRPortfolio value / loss from rating movesRating-migration probabilities, standard deviation, correlations
Market VaRMaximum market loss at a confidence and horizonRisk factors; for historical simulation, e.g. 500 days of changes
Liquidity at RiskLikely funding requirementHistoric funding amounts placed in brackets
Quick check
In a historical simulation VaR at 95% confidence, which value is read from the ordered results?

8.5 Governance of risk modelling

▼

The governance framework syllabus 8.1.3

The workbook draws on the US Federal Reserve's Supervisory Guidance on Model Risk Management. A sound framework covers (1) roles and responsibilities of the board, senior management, internal audit and external resources, and (2) policies and procedures, with a model inventory and good documentation. Strong governance is fundamental: even with satisfactory development, implementation, use and validation, weak governance reduces overall effectiveness.

Who does what syllabus 8.1.3

PartyResponsibility
Board of directorsUltimately responsible; sets the firm-wide framework including standards for development, implementation, use and validation; ensures model risk is within risk appetite and directs changes; sets the tone. Generally delegates execution and maintenance to senior management.
Senior managementEstablish policies and ensure adherence; assign competent staff with clear model ownership and manage conflicts of interest; oversee development and implementation; set model risk controls (evaluate results, ensure effective challenge, review validation and audit findings, act promptly); consider provisions for trades where the model is known to have limited capability to represent reality.
Internal auditAssesses the overall effectiveness of the framework, for individual models and in aggregate; staff need adequate modelling expertise. Staff who do day-to-day validation should not assess the overall framework (independence).
External resourcesMay help execute activities, but risk management remains internal: agree a written scope of work, have a designated internal party able to understand and evaluate external results, and keep a contingency plan if the resource becomes unavailable or unsatisfactory.
Quick check
Who is ultimately responsible for the firm-wide model risk framework?

Policies, procedures and documentation syllabus 8.1.3

Board-level policies should cover all aspects: definition of model risk, assessment of model risk, acceptable practice for development, implementation and use, appropriate validation activities, and governance and controls over the process.

  • Emphasise testing and analysis: targets for model accuracy, standards for acceptable discrepancy, and procedures to review and respond to unacceptable discrepancies.
  • Set the priority, scope and frequency of validation: how much before models go into production, and the scope of ongoing validation.
  • Back-testing for models based on history: benchmark outputs against reality once reality has caught up. A model giving VaR for one week ahead is compared with the actual market position a week later.
  • Set validation requirements for vendor and third-party models.
  • Require detailed documentation: a model inventory, results of modelling and validation, and issues and their resolution.
  • The board or its delegates periodically review the policies for consistent, rigorous practice.
Memory hook for a model risk policy: define, assess, build and use, validate, govern. Back-testing is the key validation method for historical models.

8.6 Ch 8 cheat sheet

▼

All the definitions + numbers

ItemAnswer
ModelMethod turning input data into quantitative estimates (qualitative inputs allowed if output is quantitative)
Spreadsheet arithmeticNot a model (still needs EUC care)
Model riskAdverse consequences from decisions based on incorrect or misused models
Model risk typeOperational risk
Two causesFundamental errors; incorrect or inappropriate use
BenefitsSimplified view, competitive edge, speed, accuracy, repeatability, what-if
LTCMConvergence trades, then drift to merger arbitrage; lost $4.4bn of $4.7bn in 1998; closed early 2000
Limiting assumptionsDistribution shape; past predicts future; business environment at design
Confidence-level blind spotVaR says nothing about losses beyond the cut-off; use stress testing
AI limitationsBias, data needs, opacity, cost
Op risk scenario inputsFrequency, typical loss, extreme-case severity
Op risk distributionFat-tailed, not normal; lognormal often used
Workshop extreme estimate~20 years experience = 1 in 20 = 95%
Capital severity levels1 in 200 = 99.5%; 1 in 1,000 = 99.9%
Poisson vs lognormalPoisson = frequency; lognormal = impact
99.5% of 100,00099,500th row from the bottom
Portfolio effectSum of individual 99.5% figures exceeds combined 99.5% figure
Credit VaR inputRating-migration probabilities, standard deviation, correlations
Market VaR meaningOne-day 5% VaR $1m: 0.05 probability of losing more than $1m
Historical simulationRisk factors, sample (e.g. 500 days), revalue, read 5th percentile for 95%
Governance guidanceUS Federal Reserve Supervisory Guidance on Model Risk Management
Board vs senior managementBoard owns and sets appetite; senior management executes and maintains
Internal auditAssesses framework; validators should not assess the overall framework
Back-testingCompare model output to reality once it has caught up