The first author of the paper is Mr. Shakti PC who is currently in Japan. This paper is published in Journal of Hydrology and Meteorology Nepal (Vol. 7, No. 1) on December 2010.

Published on: **Mar 3, 2016**

Published in:
Technology

- 1. Step wise multi-criteria performance evaluation of rainfall- runoff models using WETSPRO P.C. Shakti1;a;b, Shrestha Narayan Kumar1;c, Gurung Pabitra1;d1 Interuniversity Program in Water Resources Engineering (IUPWARE), Katholieke Universiteit Leuven and VrijeUniversiteit Brussel, Belgium.a Graduate School of Life and Environmental Science, University of Tsukuba, Tsukuba, Japan.b National Research Institute for Earth Science and Disaster Prevention(NIED), Tsukuba, Japan.c Department of Hydrology and Hydraulic Engineering, Vrije Universiteit Brussel, Pleinlaan 2, Brussel Belgium.d Hydro Solution Pvt. Ltd, Swayambhu, Kathmandu, NepalAbstract:This paper illustrates a methodology to evaluate model’s performance of rainfall runoffmodel using a tool called WETSPRO (Water Engineering Time Series PROcessingtool). Simulated results of physically based semi-distributed model - SWAT (Soil andWater Assessment Tool) for Kliene Nete watershed (581 km2), Belgium are consideredin this study. Paper presents a series of sequential time series processing tasks to beperformed to evaluate model’s performance thoroughly. The problem of serialdependence and heteroscedasticity is addressed and model performance evaluation ondifferent flow components (peak flows, low flows and volume) and flow volume iscarried. Performance evaluation of both flow components on their extremes is alsoperformed. Two most commonly used goodness-fit-statistics (Mean Square Error –MSE and Nash Sutcliff Efficiency − NSE) are used with number of complementarygraphical plots for evaluation propose. Results indicated model’s robust performance onpeak flows although base flows are slightly underestimated especially for lower returnperiods. Cumulative flow volumes tend to be overestimated. Based upon the study,some recommendations are summarized to enhance model’s ability to simulate theflows events.Key words: Rainfall runoff model, SWAT, WETSPRO, Kliene Nete, peak flows, lowflows. 1
- 2. 1. Introduction:Different types of rainfall-runoff models are being used by many research institutions,government organizations and other water related sectors for decision support systems.These models have to be calibrated and thoroughly validated. However, calibrating arainfall runoff model is not an easy task due to large number of model parametersinvolved especially in case of distributed rainfall runoff model (Willems, 2009).Consideration of several parameters while calibrating the rainfall runoff models tendto complicate the model calibration task and that would sometimes be highly timeconsuming (Dmitri et al., 2006). Rainfall runoff models have been calibrated usingeither a heuristic approach or using auto-calibrated algorithms. Heuristic approachmight be quite time consuming but it uses the experience and knowledge of modeler;while auto-calibrated routines are fast but might return the so called ‘optimal’parameters based on the local optimum instead of global optimum. So, combined use ofboth approaches might result quick and near global optimal parameters values to matchthe field condition. Moreover, obtaining ‘optimal’ parameter values is also a casespecific. Using a more general model structure would make the models over-parameterized and hence not identifiable. Nevertheless, calibrating rainfall runoff modelsometime requires a step wise calibration scheme as described by Willems (2010)especially for a lumped conceptual model. Traditional approaches to validate thecalibration of rainfall runoff models are based on some statistical indicators, forexample the Nash Sutcliff Efficiency – NSE (Nash and Sutcliffe, 1970). However it isdifficult to characterize the different aspect of model performance of a particular rainfallrunoff model with only one or two statistical indicator. The use of multi-objective set ofpreferably independent statistics is needed and consideration of supporting graphicalcriteria to evaluate how well the model has simulated to an events.To support performance evaluation of a rainfall-runoff model, a Water EngineeringTime Series PROcessing tool (WETSPRO) has been developed by Prof. PatrickWillems, Katholieke Universiteit Leuven, Belgium. The WETSPRO can be used toconduct flow filtering (quick and slow flow) using a numeric digital filter, to extractpeak-over-threshold (POT) values and to construct different model evaluation plots 2
- 3. (Willems, 2009). The WETSPRO follows a step wise multi-criterion performanceevaluation methodology with the use of different supplement graphs. It makes the use ofobserved and simulated results separating them into different flow components. It alsomakes sure of using independent variables by extracting nearly independent POTs. Onthe other hands, it also allows comparing cumulative volumes as well as modelperformance on more extreme events. More description about WETSPRO can be foundon Willems (2009).2. Scope of the Study:In present days, many rainfall-runoff models are extensively using for hydrologicalanalysis worldwide as well as many statistical tools are developed to evaluate the modelresults. The performance evaluation of model results is mandatory for application in thefield of water resources management. Consequently, main objective of this study focuson performance evaluation of the SWAT model results using statistical techniques,taking into account serial dependence of the model output flows and heteroscedasticityin the model residuals. Though the study is based on Belgium, it presents amethodology on how to evaluate model’s performance on better way; making itapplicable in any catchments in the world.3. Data and Methodology:Performance evaluation on the model results of a mid-sized Kliene Nete Watershed,581km2, Belgium (Figure-1) developed on Soil and Water Assessment Tool (SWAT) isillustrated in this study. SWAT is a physically-based continuous hydrological modelthat predicts the impact of land management practices on water, sediment andagricultural chemical yields in complex basins with varying soils, land use andmanagement conditions (Arnold et al., 1998; Green et al., 2008; Neitsch et al., 2002,Srinivasan et al., 1998). The model simulation results of the Kliene Nete Watershedobtained (Shrestha et al., 2010) with resulted NSE values of 74 and 67% for thecalibration (1/1/1994-12/31/1998) and validation period (1/1/1999-12/31/2002) 3
- 4. respectively is used for this study. Observed discharge data series at Grobbendonk(outlet) with daily frequency is used for the evaluation process. Figure-1: Topography of the Kleine Nete Watershed, location of the watershed is shown in the inset, (Dam et al., 2009).There are various performance evaluation indicators that mentioned in many literatures.Among all the statistical tools, the Mean Square Error (MSE) and the NSE are two mostfrequently used statistical parameters for performance evaluation of simulated results.The MSE is a measure that considers the average random discrepancy between observedriver flow discharge − qo (i) and simulated river flow discharge − qm (i) , and is given as: n 2 qm (i ) qo (i) i 1MSE (1) nWhere,qo (i) and qm (i) = the observed and simulated river discharge, respectively and,i = number of observations (1, n). 4
- 5. Of course, closer to zero MSE value is the ultimate target in case of model performanceresult, but due to much uncertainty in the model it is difficult to get desire value. TheNSE is used to quantitatively describe the accuracy of model outputs and predicativeaccuracy and is given as: n 2 q m (i ) qo (i ) i 1 MSENSE 1 n 1 2 (2) 2 S qo qo (i ) qo i 1Where, 2qo and S qo are the mean and variance of observed discharge series.But, it has been observed that the model residuals typically increase with higher flowvalues (Willems, 2009). That means higher flow values receives more weight inevaluation of statistics (1) and (2) making them more sensitive to higher flows. Hence,lower value of MSE and higher NSE can be observed provided higher peaks are nicelymatching even if the low flows (base flows) are not matching fairly. The situation islikely to be compounded because of the squared terms in the derivation of MSE andNSE. Hence, the flow values should be transformed by applying suitable transformationbefore they are used for evaluation of statistics such as (1) and (2). The Box and Coxtransformation (Box and Cox, 1964) allows in such cases. The simplicity of Box andCox transformation is that it has only one parameter ‘λ’ and this transformation is usedfor this study. The Box and Cox transformation given by: (q 1) / , if 0BC (q) (3) ln (q), if 0Where,q = Observed or simulated discharge values and,λ = Parameter of Box-Cox transformation.The Box-Cox transformation was applied to both observed and simulated river flowdischarges in the determination of the MSE and hence NSE so as to havehomoscedasticity (variances of the sampled elements in rainfall runoff modeling is not 5
- 6. varying with varying ‘q’) in model residuals. The parameter λ needs to be calibratedmaking the model residual homoscedasticity.Also, when series of observations are used, model residuals often have a serialdependence – one flow event is dependent on previous event. This dependence will behigher for a series with small time step is considered (Willems, 2009). For our case, asthe minimum time step of SWAT is one day, there will be no serial dependence foroverland flow (quick flow is divided into overland flow and interflow) because of smallcatchment with response time of around 8 hours. But for baseflow, the serialdependence is likely to affect due to longer recession time of base flow. This will causeproblems on evaluating goodness-fit-statistics (1) and (2) as the events should beindependent. This problem is addressed by extracting nearly independent POT values(Figure-2). The two subsequent events are considered as nearly independent if thefollowing three conditions are fulfilled: the time length ‘τs’ of the decreasing flank of the first event exceeds a time ‘kp’: τs > kp (4) the discharge drops down – in between the two events – to a fraction lower than ‘f’ of the peak flow: q min f (5) q max Or, close to the baseflow qbase: qmin qbase f (6) qmax the discharge increment qmax - qmin has a minimum height qlim: qmax - qmin > qlim (7)Thus, according to the procedure of selecting nearly independent POTs has threeparameters namely; kp, f and qlim. The parameter ‘kp’ can be taken equal to recessionconstant of quick flow or higher. Similarly, parameter ‘f’, in equation (5) can be takenas the upper limit of base flow fraction in the peak flow. If equation (6) is considered,then it can be taken as 5% to 15%. And, the parameter ‘qlim’ can be taken as the upperlimit of small noise peaks which needs to be avoided to be selected as POTs. It is 6
- 7. obvious that sub-flow filtering prior to POT selection will make easy to select theparameters of POT selection for example ‘kp’ and ‘f’. The sub-flow separation onWETSPRO is based on a generalization of the recursive digital filter proposed byChapman (1991) with a new filter parameter ‘w’ that represents the case-specificaverage fraction of the quick flow volumes over the total flow volumes. It is to be notedthat the original Chapman filtering has a parameter ‘k’ which is the recession constant‘k’ of the sub-flow to be separated. By this way the sub-flow filtering has twoparameters for each sub-flow. Figure-2: Parameters used in the criteria to select nearly independent POTs (Willems, 2009)Following are some equations (depicted from Willems (2009)) after the parameter ‘w’has been introduced. 7
- 8. f (t ) a1 f (t 1) a 2 (q(t ) aq(t 1)) (8)b(t ) q(t ) f (t ) ab(t 1) a3( (1 a)( f (t 1) f (t )) (9)Where ((2 v) v)a1 (10) (2 v va ) 2a2 (11) (2 v va )a3 0.5v (12) 1a exp (13) k 1 wv (14) wWith,q(t) = Total flow in time ‘t’[M3T-1]b(t) = Slow flow component in time ‘t’ [M3T-1]q(t) = Quick flow component in time ‘t’ [M3T-1]k = Recession constant [T]w =Case specific fraction of quick flow volumes over the total volume [-]4. Result and Discussion:4.1 Calibration of λ:Figure-3 is the plot of model residuals (difference between observed and simulateddischarge) and simulated discharges. It is clear that the model residuals showedheteroscedasticity as more residual variance can be observed for higher discharges. Tohave homoscedasticity in model residuals, a value of 0.25 for λ has chosen withheuristic approach. After the transformation, the model residuals evenly distributedalong ‘zero-residual’ line (thick black horizontal line). 8
- 9. 15 10 Model Residuals [m3/s] 5 0 -5 -10 Before BC-transformation After BC-transformation -15 0 5 10 15 20 25 30 35 40 45 50 Model Output [m3/s] Figure-3: Plot of SWAT model residuals with simulated discharges4.2 Flow Filtering:Filtering parameters ‘k’ and ‘w’ for baseflow separation have been chosen as 80 daysand 0.43 respectively and same for interflow are 4 days and 0.4 respectively. Hencearound 57% (1 - 0.43) of the total flow is contributed by the baseflow which is quiteobvious for relatively flat catchment having high filtering because of havingpredominant sandy soil. Estimation of those parameters has been done by visualinspection of the filter result plots especially the recession limb of the hydrograph.Figure-4 is used to estimate the baseflow recession constant by making the slant dottedline as far as possible, parallel to the trend of recession of long dry period. Figure-5 isthe plot of interflow filter result along with filtered base flow for initial 1500 to 2000days. It is to be noted that the days are counted as 0 (1/1/1994) to 3287 (12/31/2002). 9
- 10. Figure-4: Base flow filter resultFigure-5: Interflow filter result 10
- 11. 4.3 POT extraction:The parameters kp, f and qlim for extracting POTs for quick flow period are chosen as 8days, 0.6 and 3 m3/s. While the same for slow flow period are chosen as 80 days, 1 and0.5 m3/s. Since selection of POT is not really based upon any strict speculation, this isalso a trial and error process and the parameters can be well adjusted by the userdepending on the POT plots. But the methods and the parameter’s sensitivity should bewell understood. Figure-6 shows the results for the initial 1000 to 2500 days. Altogether118 POTs for quick flow and 34 POTs for slow flows are extracted. Figure-6: POT selection for quick and slow flow periods4.4 Performance evaluation plots:Table-1 shows the results of performance evaluation in terms of MSE and NSE.Looking only on these goodness-of-fit statistics it is clear that the model is quiteefficient on modelling slow flows than quick flows as indicated by higher NSE andlower MSE values for slow flow periods. 11
- 12. Table 1: Selected numbers of quick and slow flow periods with errors MSE NSE Flow Periods Number of POTs [m3/s] [%] Quick Flow 118 1.51 66 Slow Flow 34 1.12 79Figure-7 and Figure-8 are the comparison plot of peak flows (maxima) and low flows(minima) periods. Plot for maxima (Figure-7) showed relatively higher scatter butalmost zero bias (indicated by the overlapping of bisector and mean deviation line). Inan average no systematic overestimation or underestimation is observed. Plot ofminima (Figure-8) shows lower scatter but a clear negative bias (low flows or base flowcomponents are systematically underestimated). The mean ± standard deviation lines onboth plots indicate the scattering of observed and simulated values for 68% confidentinterval. Figure-7: Comparison plot of Maxima after Box-Cox Transformation ( =0.25) 12
- 13. Figure -8: Comparison on plot of Minima after Box-Cox Transformation ( =0.25)Figure-9 indicates the performance of the model in extreme high events with differentreturn periods. The plot reveals that extreme simulated discharge matches very closelyto the observed and no systematic over and underestimation is observed for thoseextremes. For lower return periods, the model performance is quite good with slightoverestimation on lower return period that indicates that the model can be used for highflow problems and applications.Figure-10 shows the performance of the model in low flow events on different returnperiods. Due to small value of low flow, it is quite difficult to analysis by puttingnormal values of low discharge in y-axis in a figure. Therefore, inverse value of lowdischarge used to see lower value of discharge in this case. For return periods of 0.5 to 5years, model tends to underestimate very low discharges. But lower discrepancies areobserved towards lower and upper tail of the empirical extreme value distribution. Socare should be taken while using the model for those events. 13
- 14. Figure 9: Comparison plot of high flow extreme value distribution 14
- 15. Figure-10: Comparison plot of low flow empirical extreme value Figure 11: Comparison plot of cumulative flow volumes (total flows)Figure-11 shows the comparison plot of cumulative discharges and it is clear that themodel tend to overestimate the total flows. This is indication of high water yieldthrough the simulation. This may be due to the effect of higher value of soil evaporationcompensation factor (ESCO) which allows less evapo-transpiration allowing substantialportion of water volume coming to the river stream. Another reason may be due to lowthreshold water depth in the shallow aquifer (GWQMN) releasing slightly greateramount of water from aquifer than it should.5. Conclusion:The paper demonstrates a methodology of model performance evaluation usingWETSPRO. As it is clear that modeling is a necessary part for the decision makers so arobust model calibration and their performance evaluation should be thoroughlychecked before the model are being used in real time. The trivial way of accessing therobustness of model output by using some goodness-of-fit statistical indicators is notalways sufficient. Therefore, consideration on problems of serial dependency and 15
- 16. heteroscedasticity should be addressed. Model performance on different flowcomponents (peak and low flows), separate evaluation of different subflows, evaluationof peak and low flow extremes and evaluation on total flow volume clearly displaysmodel’s performance.The performance evaluation made on this study is based on the calibration of SWATmodel of Kliene Nete Catchment of Belgium by Shrestha et al., (2010). It is clear thatmodel’s performance on high flows is quite satisfactory with slight overestimation onlower return periods. Performance on low flows showed slight underestimation as canbe seen from Figure-8 and 10. This indicates that the model can be used for high flowproblems but care should be taken while using it in low flow problems. In terms of totalvolume, the model has tendency to overestimate the water balance. If one uses thismodel for designing a reservoir for a larger time span, then it might be that he/she cancome up with slightly oversized reservoir design. This can be addressed bymanipulating SWAT model parameters for example; decreasing ESCO (soil evaporationcompensation factor) to allow a larger portion of water to evaporate or a higher depthfor GWQMN (threshold water depth in the shallow aquifer for flow) can be adopted soas to allow less water to release from the aquifer.6. Acknowledgement:Authors wish to express their gratitude to Prof. P. Willems and Prof. W. Bauwens fortheir immense support during study period. Special thanks go to the FlemishInteruniversity Council (VLIR) for providing the scholarship for this InternationalMaster Program in Water Resources Engineering (2007-2009) and KatholiekeUniversiteit Leuven and Vrije Universiteit Brussel, Belgium for providing the platform.Lastly, appreciate the help of PhD researcher Mr. Nirman Shrestha from KatholiekeUniversiteit Leuven, Belgium for language correction of this paper. 16
- 17. 7. References:Arnold, J.G., Srinivasan, P., Muttiah, R.S., Williams, J.R. (1998): Large area hydrologic modeling and assessment. Part I. Model development. Journal of American Water Resources Association. 34, pp. 73–89.Box, G.E.P., Cox, D.R., (1964): An analysis of transformations. Journal of the Royal Statistical Society 26, pp. 211–243.Chapman, T. (1991): Comment on Evaluation of automated techniques for base flow and recession analyses by Nathan R.J. and McMahon, T.A. Water Resources Research 27 (7), pp. 1783–1784.Dam J., Salvadore E., Van Daele T. & Batelaan, O., (2009): Case Kleine Nete: hydrologie, Wetenschappelijk rapport, NARA-2009. [INBO.R.2009.28]. Rapporten van het Instituut voor Natuur- en Bosonderzoek 2009 (28). Instituut voor Natuur- en Bosonderzoek, Brussel.Dmitri, K., George, K. and Stewart,W. F., (2006). Calibration of conceptual hydrological models revisited: 1. Overcoming numerical artefacts. Journal of Hydrology Vol. 320, pp. 173-186.Green, C.H., Griensven, A.V. (2008): Autocalibration in hydrologic modeling: Using SWAT2005 in small-scale watersheds. Environmental Modelling & Software. Elsevier Science, Vol. 23, Issue 4, pp. 422 – 434.Nash, J. E., and Sutcliffe, J. V., (1970). River flow forecasting through conceptual models part I-A discussion of principles. Journal of Hydrology, 10 (3), pp. 282– 290. 17
- 18. Neitsch, S.L., Arnold, J.G., Kiniry, J.R., Williams, J.R., and King, K.W. (2002): Soil and water assessment tool theoretical documentation. Version 2000, Texas water resources institute, Colleage station, Texas, USA.Shrestha, N.K., P.C., Shakti and Gurung, P., (2010). Calibration and validation of SWAT Model for low lying watershed: a case study on Kliene Nete Watershed, Belgium. Hydro Nepal: Journal of Water, Energy and Environment Vol. 6, 2010, pp.47-51.Srinivasan, R., Ramanarayanan, T.S., Arnold, J.G. and. Bednarz, S.T. (1998): Large area hydrological modeling and assessment. Part II: Model application. Journal of American Water Resources Association, 34(1), pp. 91-101Willems, P., (2009). A time series tool to support the multi-criteria performance evaluation of rainfall-runoff models. Environmental Modelling and Software 24(3), pp. 311-321.Willems, P., (2010). VHM approach: transparent, step-wise and data mining based identification and calibration of parsimonious lumped conceptual rainfall-runoff models. Journal of Hydrology (submitted). 18