Please wait a minute...
Tsinghua Science and Technology  2019, Vol. 24 Issue (06): 706-715    doi: 10.26599/TST.2018.9010092
Residuals-Based Deep Least Square Support Vector Machine with Redundancy Test Based Model Selection to Predict Time Series
Yanhua Yu*, Jie Li
∙ Yanhua Yu and Jie Li are with the School of Computer, Beijing University of Posts and Telecommunications, Beijing 100876, China. E-mail:
Download: PDF (850 KB)      HTML
Export: BibTeX | EndNote (RIS)      


In this paper, we propose a novel Residuals-Based Deep Least Squares Support Vector Machine (RBD-LSSVM). In the RBD-LSSVM, multiple LSSVMs are sequentially connected. The second LSSVM uses the fitting residuals of the first LSSVM as input time series, and the third LSSVM trains the residuals of the second, and so on. The original time series is the input of the first LSSVM. Additionally, to obtain the best hyper-parameters for the RBD-LSSVM, we propose a model validation method based on redundancy test using Omni-Directional Correlation Function (ODCF). This method is based on the fact when a model is appropriate for a given time series, there should be no information or correlation in the residuals. We propose the use of ODCF as a statistic to detect nonlinear correlation between two random variables. Thus, we can select hyper-parameters without encountering overfitting, which cannot be avoided by only cross validation using the validation set. We conducted experiments on two time series: annual sunspot number series and monthly Total Column Ozone (TCO) series in New Delhi. Analysis of the prediction results and comparisons with recent and past studies demonstrate the promising performance of the proposed RBD-LSSVM approach with redundancy test based model selection method for modeling and predicting nonlinear time series.

Key wordstime series prediction      information redundancy      residuals-based deep Least Squares Support Vector Machine (LSSVM)      Omni-Directional Correlation Function (ODCF)     
Received: 10 January 2018      Published: 15 March 2018
Corresponding Authors: Yanhua Yu   
About author:

Jie Li received the BEng degree from Shenyang University of Aviation and Aerospace in 1999, MEng degree from Sichuan University in 2004, and PhD degree from Beijing University of Posts and Telecommunications in 2009. He has published more than 10 papers. He is now a lecturer with the School of Computer at Beijing University of Posts and Telecommunications. His research interests include cloud computing, mobile network, and data mining.

Cite this article:

Yanhua Yu, Jie Li. Residuals-Based Deep Least Square Support Vector Machine with Redundancy Test Based Model Selection to Predict Time Series. Tsinghua Science and Technology, 2019, 24(06): 706-715.

URL:     OR

Fig. 1 RBD-LSSVM with two layers.
1st layer182446
2nd layer1131014
3rd layer1114
4th layer2411–394 00140.3–0.55
Table 1 Hyper-parameter values for the four-layer RBD-LSSVM validated using ODCF test.
Fig. 2 Predicted and target test series of annual sunspot number for years 1956–1979.
MethodNMSE (%)
Local LSSVM[4]7.8
Neural network[21]15.1
Table 2 Performance comparison for sunspot number series prediction from 1956 to 1979.
Fig. 3 ACF for original monthly TCO in New Delhi.
Fig. 4 ACF for seasonally differenced monthly TCO in New Delhi.
1st layer3041650149
2nd layer11216
3rd layer13612
4th layer8131–105117
Table 3 Hyper-parameter values for the four-layer RBD-LSSVM validated using ODCF test.
Fig. 5 Target and predicted test series for TCO in New Delhi (48 one-step-ahead predictions).
MethodRMSEPearson correlation coefficientNMSE
Iterative error correction based on direct search in Ref. [7]8.420.800.3810
ϵ?-?S?V?M in Ref. [9]10.30.680.5054
Table 4 Performance comparison for total ozone column in New Delhi.
[1]   Box G. E. P., Jenkins G. M., and Reinsel G. C., Time Series Analysis: Forecasting and Control. Beijing, China: Posts & Telecom Press, 2005.
[2]   Chen H. X., Feng S., Pei X., Zhang Z., and Yao D. Y., Dangerous driving behavior recognition and prevention using an autoregressive time-series model, Tsinghua Science and Technology, vol. 22, no. 6, pp. 682-690, 2017.
[3]   Wang J., Tang L. Y., and Luo Y. Y., A weighted EMD-based prediction model based on TOPSIS and feed forward neural network for noised time series, Knowledge Based System, vol. 132, pp. 167-178, 2017.
[4]   Arash M. and Majid A., Developing a local least-squares support vector machines-based neuro-fuzzy model for nonlinear and chaotic time series prediction, IEEE Transactions on Neural Networks and Learning System, vol. 24, no. 2, pp. 207-218, 2013.
[5]   Han M. and Xu M. L., A hybrid prediction model of multivariate chaotic time series based on error correction, Acta Phys. Sin., vol. 62, no. 12, p. 120510, 2013.
[6]   Guo Z. Q., Wang H. Q., and Liu Q., Financial time seriesforecasting using LPP and SVM optimized by PSO, Soft Computing, vol. 17, no. 5, pp. 805-818, 2013.
[7]   Tang Z. J., Ren F., Peng T., and Wang W. B., A least square support vector machine prediction algorithm for chaotic time series based on the iterative error correction, Acta Phys. Sin., vol. 63, p. 050505, 2014.
[8]   Sapankevych N. I. and Sankar R., Time series prediction using support vector machines: A survey, IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 24-38, 2009.
[9]   Wang G. L., Yang P. C., and Mao Y. Q., On the application of non-stationary time series prediction based on the SVM method, Acta Phys. Sin., vol. 57, no. 2, pp. 714-719, 2009.
[10]   Yu Y. H. and Song J. D., Redundancy-test-based hyperparameters selection approach for support vector machines to predict time series, Acta Phys. Sin., vol. 61, no. 17, p. 170516, 2012.
[11]   Chen T. T. and Lee S. J., A weighted LS-SVM based learning system for time series forecasting, Information Sciences, vol. 299, pp. 99-116, 2015.
[12]   Zhu Q. M. and Zhang L. F., Development of omni-directional correlation functions for nonlinear model validation, Automatica, vol. 43, no. 9, pp. 1519-1531, 2007.
[13]   Zhang L. F., Zhu Q. M., and Longden A., A correlation-test-based validation procedure for identified neural networks, IEEE Trans. on Neural. Netw., vol. 20, no. 1, pp. 1-13, 2009.
[14]   Vapnik V. N., The Nature of Statistical Learning Theory, 2nd Ed. New York, NY, USA: Springer, 1999.
[15]   Deng N. Y. and Tian Y. J., Support Vector Machines: Theory, Algorithm and Extension (in Chinese). Beijing, China: Science Press, 2009.
[16]   Suykens J. A. K., Gestel T. V., Brabanter J. D., Moor B. D., and Vandewalle J., Least squares support vector machines, International Journal of Circuit Theory & Applications, vol. 27, no. 6, pp. 605-615, 2002.
[17]   Ljung L., System Identification: Theory for the User, 2nd Ed. Prentice-Hall Inc., 1999.
[18]   Mao K. Z. and Billings S. A., Multi-directional model validity tests for non-linear system identification, International Journal of Control, vol. 73, no. 1, p. 32, 2000.
[19]   Brock W. A., Dechert W. D., Scheinkman J., and LeBaron B., A test for independence based on the correlation dimension, Econometric Reviews, vol. 15, pp. 197-235, 1996.
[20]   Zhou Z. H., Ensemble Methods: Foundations and Algorithms. Taylor & Francis, 2012.
[21]   Yilmaz S. and Oysal Y., Fuzzy wavelet neural network models for prediction and identification of dynamical systems, IEEE Trans. Neural Netw., vol. 21, no. 10, pp. 1599-1609, 2010.
[22]   Cao L. J., Support vector machines experts for time series forecasting, Neurocomputing, vol. 51, pp. 321-339, 2003.
[23]   Weigend A. S., Huberman B. A., and Rumelhart D. E., Predicting the future: A connectionist approach, Int. J. Neural Systems, vol. 1, pp. 193-209, 1990.
[24]   Tong H. and Lim K. S., Threshold autoregressive, limit cycles and cyclical data, Journal of Royal Statistic Society, vol. 42, pp. 245-292, 1980.
[1] Yu Liu, Shuting Dong, Mingming Lu, Jianxin Wang. LSTM Based Reserve Prediction for Bank Outlets[J]. Tsinghua Science and Technology, 2019, 24(1): 77-85.