Presentation on ‘Operational Risk Loss Forecasting Model for Stress Testing – A Three-Stage Approach’ made by Dr. James Lu, Director, Risk & Analytics, CRISIL Global Research & Analytics (GR&A) at The 17th Annual OpRisk North America 2015, New York

Published on: **Mar 3, 2016**

- 1. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting Model for Stress Testing A Three-Stage Approach James Lu The 17th Annual OpRisk North America 2015, New York March 2015
- 2. ©2014CRISILLtd.Allrightsreserved. Agenda Operational Risk Loss Forecasting for Stress Testing – Regulatory Expectation and Challenges – Industry Practice and Methodology Rebuilding the Operational Risk Loss Forecasting Model – Background and Challenges – A Three-Stage Approach – ARIMAX Model to Project External Losses – OLS Model to Project Internal Losses – Simulation to Estimate Tail Risk Using Scenario Analysis – The Model Results – Conclusions and Limitations 2
- 3. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting for Stress Testing Regulatory Background and Expectation 3 The Fed’s CCAR requires banks to have internal capital planning processes – Operational risk loss is a component of PPNR, which is the first building block of the CCAR process Operational risk loss estimation methodologies vary among banks and so do the Fed’s expectations The Fed has given guidance regarding estimation methodologies – Stronger practices vs. lagging practices Banks should use internal op risk loss data as a starting point to provide historical perspective – They can then incorporate forward-looking elements, idiosyncratic risks, and tail events to estimate losses for the CCAR projection window Operational Risk Loss Forecasting Challenges Limited internal loss history Link loss history with macroeconomic drivers, including bank-specific risk drivers Quantify extreme severe but plausible loss events
- 4. ©2014CRISILLtd.Allrightsreserved. 4 Operational Risk Loss Forecasting for CCAR Industry Practice and Challenges Operational Risk Loss Estimation Flowchart Banks’ approaches vary in each of the following steps due to data richness and business sophistication. A few banks conduct benchmark analysis. Internal loss historical database, supplemented by external historical loss Macroeconomic variables Scenario analysis add-on to capture tail risk which internal loss history may have not experienced Analytical models or judgment-based estimation to predict future ops risk losses for the baseline scenario Analytical models or judgment-based estimation to predict future ops risk losses for stressed scenarios and bank idiosyncratic risk Management overlay for legal/litigation/compliance risk and other risks Benchmarks to either support or challenge the loss projections
- 5. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting for CCAR Industry Practice and Challenges Common operational risk loss estimation methodologies Modeling Approaches – Regression model using macroeconomic variables – The ARIMA/ARIMAX model to capture the auto regressive nature of loss history – Modified loss distribution approach – Scenario analysis to capture tail risk Criteria for Selecting the Final Model The selected modeling approach should have the following characteristics: – Segmentation needs to be granular enough given the data availability – Model design and outcomes should be empirically robust and statistically sound – Model performance needs to exhibit reasonable levels of stability and accuracy – Scenario analysis add-on and management overlay should be fully reviewed, challenged and well documented 5
- 6. ©2014CRISILLtd.Allrightsreserved. Agenda Operational Risk Loss Forecasting for Stress Testing – Regulatory Expectation and Challenges – Industry Practice and Methodology Rebuilding the Operational Risk Loss Forecasting Model – Background and Challenges – A Three-Stage Approach – ARIMAX Model to Project External Losses – OLS Model to Project Internal Losses – Simulation to Estimate Tail Risk Using Scenario Analysis – The Model Results – Conclusions and Limitations 6
- 7. ©2014CRISILLtd.Allrightsreserved. Modeling Challenges Rebuilding the Ops Risk Loss Forecasting Model Background and Challenges 7 The bank has less than four years (i.e.15 quarters) of operational risk data in its database, which reliably and consistently captures operational risk loss dates, loss events, lines of business, recoveries, etc. ORX ops-risk database has quarterly loss data since 2002. How do we link external loss history with internal loss history? What is the granularity of modeling, given both the richness of external loss data and short history (and data sparseness) of internal loss data? Literature and industry experience indicate a lack of strong predictors for ops-risk loss model, so developing statistically sound and empirically robust models is a challenge Quantitatively-driven scenario analysis add-on as a tail risk is still evolving and therefore a challenge for most banks to factor it into their models New Modeling Approach The newly developed three-stage model and quantitatively-driven scenario analysis respond to the challenges faced by both the bank and the industry
- 8. ©2014CRISILLtd.Allrightsreserved. Three-Stage Modeling Methodology 8 The three-stage model utilizes both external and limited internal ops-risk loss data to forecast the future nine quarters’ (9Q) ops-risk loss for the bank in both the baseline and stress scenarios Management overlay is added to capture legal/litigation/compliance risk The implemented model meets the regulatory guidelines for stronger practices regarding ops-risk loss forecasting methodologies Ops-Risk Loss Forecasting Model Stage 1 Stage 3Stage 2 The ARIMAX model to forecast future 9Q external ops-risk loss by geography and LoB using macroeconomic variables The OLS model to forecast future 9Q internal ops-risk loss by geography and LoB using projected external quarterly loss from Stage 1 and macroeconomic/bank idiosyncratic risk drivers Simulate scenario analysis results to capture tail risk that was not experienced in internal loss history
- 9. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting for CCAR 9 Three-Stage Modeling Process Flowchart The ARIMAX model to predict 9Q ops-risk losses for CCAR horizon by geography and LoB ORX Loss History Macro Variables Projected external losses by geography and LoB Mapped Macro Variables Mapped Bank Specific Variables Bank Internal Loss History Projected 9Q of internal losses by geography and LoB The regression model to predict 9Q ops-risk losses for CCAR horizon by geography and LoB Scenario Analysis by Event Type 1 Scenario Analysis by Event Type 2 Scenario Analysis by Event Type 3 Projected 9Q internal losses with tail risk Projected 9Q internal losses with tail risk and management overlay Stage 1 Stage 2 Stage 3
- 10. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting for CCAR Stage 1: ARIMAX Model Using External ORX Data (1/3) 10 The ARIMAX model to predict 9Q ops-risk losses for CCAR horizon by Geography and LoB ORX Loss History Macro Variables Projected external losses by geography and LoB Map the bank’s portfolio to ORX data structure by geography and LoB. There are six geography and LoB combinations. This mapping is based on the assumption that the bank and other banks in ORX have similar risk drivers by geography and LoB combination Data integrity and cleansing include data series normalization, missing data and outlier treatment, specifically quarterly external loss is normalized by the number of banks in the ORX. Variable transformations are tested, and the selected dependent variable is the logarithm of quarter-over- quarter change of ops-risk loss per bank The ARIMAX model offers the following benefits: – Captures the relationship between observations across time – Estimates relationships and produces forecasts that utilize both the information in past values of the series (quarterly loss) and the information contained in independent variables (macroeconomic variables) – Methodology is mature and easy to diagnose and estimate – ORX data series has 12 years of data and covers economic cycles The ARIMAX model has one limitation: – In its data history, ORX data does not capture the bank’s portfolio size, which could be a driver GEO 1 – Retail GEO 1 – Commercial GEO 2 – Retail GEO 2 – Commercial GEO 3 – Retail GEO 1 – Corporate & Others Bank (Geography & LoB) GEO 1 GEO 2 GEO 3 GEO 4 GEO 5 ORX Data Geography LoB Retail Commercial Corporate & Others
- 11. ©2014CRISILLtd.Allrightsreserved. 𝜆 𝑡,𝑝 𝑂𝑅𝑋 = 𝑙𝑜𝑔 𝐿 𝑡,𝑝 𝑂𝑅𝑋 𝐿 𝑡−1,𝑝 𝑂𝑅𝑋 = 𝑙𝑜𝑔𝐿 𝑡,𝑝 𝑂𝑅𝑋 − 𝑙𝑜𝑔𝐿 𝑡−1,𝑝 𝑂𝑅𝑋 = 𝑓 𝑥 𝑝, 𝜙 𝑝, 𝜃 𝑝 Operational Risk Loss Forecasting for CCAR Stage 1: ARIMAX Model Using External ORX Data (2/3) 11 The ARIMAX model specification – AR: Autoregressive – Time series is a function of its own past – MA: Moving Average – Time series is a function of past “shocks”(deviations, innovations, errors, etc.) – I: Integrated – Differencing provides stochastic trend and seasonal components, so forecasting requires integration or undifferencing – X: Exogenous – Time series is influenced by external factors The selected dependent variable is a logarithm of QoQ ops-risk loss ratios (𝜆 𝑡,𝑝 𝑂𝑅𝑋 ) for a proxy of geography and LoB combination, and it measures the relative change of ops-risk loss. It can be modeled as a function of macroeconomic variables 𝑥 𝑝 along with lagged values of ORX log- returns (ϕ) and the lagged forecast errors (θ) which are specific to that proxy (i.e.𝑓 𝑥 𝑝, 𝜙 𝑝, 𝜃 𝑝 The ARIMAX model combines time-series analysis and regression analysis. Hence, in many cases, it can produce better forecasts than using either technique alone. The ARIMAX model includes a structural (economic) explanation for the part of the variance of the dependent variable that can be explained structurally while explaining the part of the variance that cannot be explained structurally with a time-series model. We choose the best model based on – Low AIC – Low MAPE – Statistical significance of macroeconomic variables, MA, and AR terms The ARIMAX model to predict 9Q ops-risk losses for CCAR horizon by geography and LoB ORX Loss History Macro Variables Projected external losses by geography and LoB
- 12. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting for CCAR Stage 1: The ARIMAX Model Using External ORX Data (3/3) 12 The model outcomes indicate that all macroeconomic and structural variables are significant and economically explainable The same model is used to project 9Q external losses for baseline and stress scenarios The ARIMAX model to predict 9Q ops-risk losses for CCAR horizon by geography and LoB ORX Loss History Macro Variables Projected external losses by geography and LoB GEO 1 – Retail Portfolio Conditional Least Squares Estimation Parameter Estimate Standard Error t Value Approx Pr > |t| Lag Variable MA1,1 -0.47548 0.15491 -2.88 0.0066 1 log_loss_ratio_Rtl VAR1 6.84435 3.19104 2.68 0.0110 0 CPI_SA_YY_Q8 VAR2 1 GEO 1 – Commercial Portfolio 3.02520 4.12118 3.65 0.0008 0 Unemp_YY_Abs_Q4 VAR4 0.25252 0.05616 3.78 0.0005 0 GDP_QQ_Rel_Q8 Conditional Least Squares Estimation Parameter Estimate Standard Error t Value Approx Pr > |t| Lag Variable MA1,1 -1.15427 0.43507 -2.75 0.0100 1 log_loss_ratio_Cml MA1,2 -1.28583 0.30382 -3.57 0.0012 2 log_loss_ratio_Cml MA1,3 -1.51444 0.32225 -4.13 0.0003 3 log_loss_ratio_Cml AR1,1 -1.12522 0.37971 -2.81 0.0084 1 log_loss_ratio_Cml AR1,2 -0.78412 0.23962 -3.26 0.0027 2 log_loss_ratio_Cml AR1,3 -1.021041 0.24620 -4.07 0.0003 3 log_loss_ratio_Cml VAR1 27.27729 12.40279 2.04 0.0502 0 Unemp_QQ_Abs_Q6 VAR2 -0.0008847 0.0004363 -2.27 0.0300 0 GDP_QQ_Abs_Q2 VAR3 0.0552E-7 9.5156E-8 -2.18 0.0369 0 Pers_Bk_Filings_YY_Abs_$_Q4
- 13. ©2014CRISILLtd.Allrightsreserved. 13 Projected external losses by geography and LoB Mapped Macro Variables Mapped Bank Specific Variables Bank Internal Loss History Projected 9Q internal losses by geography and LoB Regression model to predict 9Q ops-risk losses for CCAR horizon by geography and LoB Operational Risk Loss Forecasting for CCAR Stage 2: Regression Model for the Bank’s Internal Loss Forecasting (1/2) The Stage 2 model uses the forecasted external losses from the ARIMAX model in Stage 1, along with macroeconomic and bank-specific risk factors to forecast the bank’s internal losses. The regression model was selected over the ARIMAX model due to the short history of internal loss data. Internal loss forecast model is specified as 𝐿 𝑡,𝑝 𝐼𝑛𝑡 = 𝐿 + 𝑥 + 𝜆 𝑡,𝑝 𝑂𝑅𝑋 That is, the bank’s internal loss is a function of the average quarterly historical loss 𝐿, macro and bank- specific variables 𝑥 (including balance sheet size), and the forecasted values of the ORX losses, 𝜆 𝑡,𝑝 𝑂𝑅𝑋 It is worthwhile to note that – We include forecasted external loss as an internal loss driver. This is based on the assumption that the bank and other banks in ORX share similar operational risk loss trends – We include the bank’s internal loss average in the regression model on the assumption that the historical average is the starting point of future loss. In other words, future loss will be around the historical average, adjusted by forecasted external loss, other risk factors, and balance sheet size
- 14. ©2014CRISILLtd.Allrightsreserved. 14 Projected external losses by geography and LoB Mapped Macro Variables Mapped Bank Specific Variables Bank Internal Loss History Projected 9Q internal losses by geography and LoB Regression model to predict 9Q ops--risk losses for CCAR horizon by geography and LoB Operational Risk Loss Forecasting for CCAR Stage 2: Regression Model for the Bank’s External Loss Forecasting (2/2) The model outcomes indicate that the identified risk drivers are significant and economically explainable The same model is used to project the baseline scenario and all stress scenarios for 9Q of the bank’s internal losses GEO 1 Commercial Internal OLS Model (Adjusted R 2 = 0. 52) Parameter Estimates Variable Label DF Parameter Estimate Standard Error t Value Pr > |t| Tolerance V.I CPI_SA_YY_Q4 1 -176.0824 46.56550 -4.04 0.0014 0.99499 1.0 log_lossratio_hat Forecast for log_dloss_sm4 1 14.6227 7.06965 2.29 0.0397 0.99499 1.0 Analysis Variable : MAPE Minimum Mean Median Maximum 0.0205061 1.7752406 0.8452006 9.2587490 GEO 1Retail Internal OLS Model (Adjusted R2 = 0.852) Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > |t| Tolerance Variance Inflation Intercept 1 15.20999 2.94609 5.51 0.0003 . 0 Auto_Sales_NSA 1 2.42056 0.46628 4.60 0.0010 0.18560 5.38782 HPI 1 -0.10252 0.02193 -4.38 0.0014 0.61482 1.62650 Loglossratio_hat_Q3 1 -0.84000 0.31150 -2.66 0.0239 0.53596 1.86580 Analysis Variable :MAPE Minimum Mean Median Maximum 0.0258840 0.2257054 0.1640946 0.8851183
- 15. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting for CCAR Performance Testing 15 Model performance testing was conducted to compare the new model with the previous years’ models on both model stability (Adjusted R2) and predicting accuracy (Mean Absolute Percent Error, or MAPE*) The new model is significantly better than the previous model on both stability and accuracy. In addition, the new model more appropriately reflects the risk characteristics of ops-risk and is better aligned with the supervisory guidance of operational risk loss forecasting methodologies for CCAR The Fed examiner’s comment was positive: “Model better designed and performance improved significantly.” * Mean Absolute Percent Error (MAPE): Absolute (Forecast – Actual) / Actual ** Not based on the same model specification, therefore not a direct comparison Model Performance Comparison – Stability** Model Performance Comparison – Accuracy Geography & LoB Prior Model New Model R2 Adj. R2 R2 Adj. R2 GEO 1 Retail 0.18 0.13 0.88 0.85 GEO 1 - Commercial 0.21 0.14 0.55 0.52 Geography & LoB Prior Model New Model Min Mean Max Min Mean Max GEO 1 Retail 12% 136% 892% 2.30% 23% 89% GEO 1 - Commercial 2% 359% 6629% 2.00% 178% 926%
- 16. ©2014CRISILLtd.Allrightsreserved. 16 Projected 9Q internal losses by geography and LoB Scenario Analysis by Event Type 1 Scenario Analysis by Event Type 2 Scenario Analysis by Event Type 3 Projected 9Q internal losses with tail risk Projected 9Q internal losses with tail risk and management overlay The Stage 3 of the model involves simulating scenario analysis results to capture ops-risk tail risk, which are the frequency and severity that current internal loss history has not experienced. Scenario analysis is based on the seven Basel II AMA ops-risk loss event categories*. A group of SMEs discussed and selected certain categories that are specifically applicable to the bank’s exposure and loss experience For the selected categories below, SMEs determined the “Most Likely” and “Worst Plausible” loss frequency and severity We use Poisson distribution to simulate loss distribution to select certain percentiles as tail risk losses under extreme scenarios. We select the median as the baseline and 55 percentile for adverse and 60 percentile as the severely adverse and BHC stress scenarios (e.g. a scalar of 118% is applied to the adverse scenario over the baseline scenario for each of the 9Q in the projection window) Operational Risk Loss Forecasting for CCAR Stage 3: Simulation of Scenario Analysis for Ops Risk Tail Risk * The seven loss event categories are internal fraud; external fraud; employment practices and workplace safety; clients, products and business practices; damage to physical assets; business disruption and system failure; and execution, delivery and process management LoB Event Category M.L Freq. M.L. Sev. M.L. Loss W.P. Freq. W.P. Sev. W.P. Loss W.P./M.L. Ratio GEO 1 Commercial CPBP - Securitization Breach 10 0.25 2.5 0.0067 5000 33.5 13.4 GEO 1 Retail Internal Fraud 1500 0.0001 0.15 0.02 500 10 66.67 Corporate Treasury EDPM 1 2.5 2.5 0.02 200 4 1.60 Corporate IT BDSF - Outage 60 0.01 0.6 0.002 50000 100 166.67 LoB Event Category 45%ile Median 55%ile 60%ile 65%ile GEO 1 - Commercial CPBP - Securitization Breach 84% 100% 121% 145% 175% GEO 1 - Retail Internal Fraud 93% 100% 106% 117% 142% Corporate Treasury EDPM 75% 100% 132% 168% 215% Corporate IT BDSF - Outage 90% 100% 113% 129% 151% Average 86% 100% 118% 140% 171%
- 17. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting for CCAR Ops-Risk Loss Forecasting Results (Bank Total Ops Risk Loss) 17 The bank’s ops-risk loss forecasting for the baseline scenario and three stress scenarios Tail risk scalars of 118% and 140% are applied to adverse and BHC stress/severe adverse scenarios, respectively, over the baseline loss projections Finally, management overlay of $150mn (scaled) is added to the three stress scenarios to cover legal/litigation/compliance risk for the projection window (not included in the graphs below) 0 20 40 60 80 100 120 The Bank Ops Risk Loss Forecasting Baseline Adverse BHC Stress Sev. Adverse 0 20 40 60 80 100 120 140 160 180 The Bank Ops Risk Loss Forecasting with Tail Risk Baseline Adverse BHC Stress Sev. Adverse
- 18. ©2014CRISILLtd.Allrightsreserved. Operational Risk Loss Forecasting for CCAR Conclusion and Limitations 18 The link between Stage 1 and Stage 2 of the model is based on the assumption that the bank’s ops-risk losses and other banks’ losses have similar loss drivers within the same geography and line of business Historical average as the starting point for internal loss projection should be on the conservative side and also in alignment with regulatory expectations (historical perspective view) Scenario creation is not based on quantitative analysis, but rather based on forward- looking of stress scenarios pertinent to the bank’s portfolio and also based on known loss experience Tail risk scalars are judgment-based This ops-risk loss forecasting does not incorporate internal control (e.g. Risk Control Self Assessment or RCSA) and other risk indicators into the estimation methodology. However, a few banks have incorporated these into their estimation methodologies Benchmark and sensitivity analysis have not been conducted, but they are Fed expectations in the long run
- 19. ©2014CRISILLtd.Allrightsreserved. www.CRISIL.com/gra ©2014CRISILLtd.Allrightsreserved.