Abstract
In this work, a novel method called epsilon-nonparallel support vector regression (ε-NPSVR) is proposed. The reasoning behind the nonparallel support vector machine (NPSVM) method for binary classification is extended for predicting numerical outputs. Our proposal constructs two nonparallel hyperplanes in such a way that each one is closer to one of the training patterns, and as far as possible from the other. Two epsilon-insensitive tubes are also built for providing a better alignment for each hyperplane with their respective training pattern, which are obtained by shifting the regression function up and down by two fixed parameters. Our proposal shares the methodological advantages of NPSVM: A kernel-based formulation can be derived directly by applying the duality theory; each twin problem has the same structure of the SVR method, allowing the use of efficient optimization algorithms for fast training; it provides a generalized formulation for twin SVR; and it leads to better performance compared with the original TSVR. This latter advantage is confirmed by our experiments on well-known benchmark datasets for the regression task.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abaszade M, Effati S (2018) Stochastic support vector regression with probabilistic constraints. Appl Intell 48(1):243–256
Alcalá-Fdez J, Fernandez A, Luengo J, Derrac J, García S, Sánchez L, Herrera F (2011) Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. Journal of Multiple-Valued Logic and Soft Computing 17(2-3):255–287
Bache K, Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml
Balasundaram S, Meena Y (2016) Training primal twin support vector regression via unconstrained convex minimization. Appl Intell 44(4):931–955
Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:27:1–27:27. software available at http://www.csie.ntu.edu.tw/cjlin/libsvm
Chen X, Yang J, Liang J, Ye Q (2012) Smooth twin support vector regression. Neural Comput & Applic 21:505–513
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273–297
Demšar J (2006) Statistical comparisons of classifiers over multiple data set. J Mach Learn Res 7:1–30
Deng N, Tian Y, Zhang C (2012) Support vector machines: optimization based theory, algorithms, and extensions. Chapman and Hall/CRC, Boca Raton
Drucker H, Burges C, Kaufman L, Smola A, Vapnik V (1997) Support vector regression machines. In: Advances in neural information processing systems (NIPS), vol 9. MIT Press, pp 155–161
Gorban A, Tyukin I, Prokhorov D, Sofeikov K (2016) Approximation with random bases: Pro et contra. Inf Sci 364:129–145
Jayadeva Khemchandani R, Chandra S (2007) Twin support vector machines for pattern classification. IEEE Trans Pattern Anal Mach Intell 29(5):905–910
Jiang H, Zhang Y, Muljadi E, Zhang J, Gao DW (2018) A short-term and high-resolution distribution system load forecasting approach using support vector regression with hybrid parameters optimization. IEEE Trans Smart Grid 9(4):3341–3350
Khemchandani R, Goyal K, Chandra S (2016) Twsvr: regression via twin support vector machine. Neural Netw 74:14–21
López J, Maldonado S (2018) Robust twin support vector regression via second-order cone programming. Knowl-Based Syst 152:83–93
López J, Carrasco M, Maldonado S (2017) A robust formulation for twin multiclass support vector machine. Appl Intell 47(4):1031–1043
Maldonado S, López J (2017) Synchronized feature selection for support vector machines with twin hyperplanes. Knowl-Based Syst 132:119–128
Maldonado S, Weber R (2010) Feature selection for support vector regression via kernel penalization. In: Proceedings of the 2010 International Joint Conference on Neural Networks, Barcelona, Spain, pp 1973–1979
Maldonado S, López J, Carrasco M (2016) A second-order cone programming formulation for twin support vector machines. Appl Intell 45(2):265–276
Mangasarian OL, Musicant DR (1999) Successive overrelaxation for support vector machines. IEEE Trans Neural Netw 10(5):1032–1037. https://doi.org/10.1109/72.788643
Melki G, Kecman V, Ventura S, Cano A (2018) Ollawv: online learning algorithm using worst-violators. Appl Soft Comput 66:384–393
Peng X (2010) Tsvr: an efficient twin support vector machine for regression. Neural Netw 23(3):365–372
Peng X (2012) Efficient twin parametric insensitive support vector regression model. Neurocomputing 79:26–38
R Rastogi R, Ananda P, Chandra S (2017) L1-norm twin support vector machine-based regression. Optimization 66(11):1895–1911
Sapankevych N, Sankar R (2009) Time series prediction using support vector machines: a survey. IEEE Comput Intell Mag 4:24–38
Scardapane S, Wang D (2017) Randomness in neural networks: an overview. WIREs Data Min Knowl Discovery, vol 7(2). https://doi.org/10.1002/widm.1200
Schölkopf B, Smola AJ (2002) Learning with Kernels. MIT Press, Cambridge
Schölkopf B, Smola AJ, Williamson RC, Bartlett PL (2000) New support vector algorithms. Neural Comput 12(5):1207–1245
Shao YH, Zhang CH, Yang ZM, Jing L, Deng N (2013) An epsilon-twin support vector machine for regression. Neural Comput & Applic 23:175–185
Singh M, Chadha J, Ahuja P, Jayadeva Chandra S (2011) Reduced twin support vector regression. Neurocomputing 74:1474–1477
Sturm J (1999) Using sedumi 1.02, a matlab toolbox for optimization over symmetric cones. Optim Methods Softw 11(12):625–653. special issue on Interior Point Methods (CD supplement with software)
Tang L, Tian Y, Yang C (2018) Nonparallel support vector regression model and its smo-type solver. Neural Nertworks 105:431–446
Tanveer M, Shubham K, Aldhaifallah M, Nisar K (2016) An efficient implicit regularized lagrangian twin support vector regression. Appl Intell 44(4):831–848
Taylor JW, Buizza R (2002) Neural network load forecasting with weather ensemble predictions. IEEE Trans Power Syst 17(3):626–632
Tian Y, Qi Z, Ju X, Shi Y, Liu X (2014a) Nonparallel support vector machines for pattern classification. IEEE Transactions on Bybernetics 44(7):1067–1079
Tian Y, Zhang Q, Liu D (2014b) nu-nonparallel support vector machine for pattern classification. Neural Comput & Applic 25(5):1007–1020
Tian YJ, Ju XC (2015) Nonparallel support vector machine based on one optimization problem for pattern recognition. Journal of the Operations Research Society of China 3(4):499–519
Vapnik V (1998) Statistical learning theory. Wiley, New Jersey
Wang D, Li M (2017) Stochastic configuration networks: fundamentals and algorithms. IEEE Trans on Cybernetics 47(10):3466–3479
Wang H, Shi Y, Niu L, Tian Y (2017) Nonparallel support vector ordinal regression. IEEE Transactions on Cybernetics 47(10):3306–3317
Zhao J, Xu Y, Fujita H (2019) An improved non-parallel universum support vector machine and its safe sample screening rule. Knowl-Based Syst 170:79–88
Acknowledgements
This research was partially funded by CONICYT, FONDECYT projects 1160894 and 1160738, and by the Complex Engineering Systems Institute (CONICYT, PIA, FB0816).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: Proof of proposition 1
Proof
We note that the twin problems (15)–(16) are convex with a Slater point, and thus, the Karush-Kuhn-Tucker (KKT) conditions are necessary and sufficient for deriving the dual form. Only the proof for Formulation (15) is provided; the same reasoning can be followed for obtaining the dual formulation of Problem (16).
The Lagrangian function associated to Problem (15) is given by
where \(\boldsymbol {\alpha }_{1},\boldsymbol {\alpha }_{1}^{*},\boldsymbol {\beta }_{1},\boldsymbol {\beta }_{1}^{*},\boldsymbol {\gamma }_{1},\boldsymbol {\gamma }_{1}^{*}\in \Re ^{m}_{+}\) denote the Lagrange multipliers. This function can be rewritten as
Then, the KKT conditions for Formulation (15) are given by
By using (28)–(32) in (27), one has
On the other hand, from (30), (31), (32) and (36) it follows that
Hence, taking into account the relations (41) and (42), we obtain the dual formulation of problem (15)
In the same way, the dual formulation of problem (16) is given by
□
Appendix B: Proof of proposition 2
Proof
- (a)
The first part follows from (28). The other follows from the KKT conditions of problem (16).
- (b)
Let us suppose that α1i > 0, then, from (37) we obtain
$$ y_{i}=(\mathbf{A}\mathbf{w}_{1})_{i}+b_{1}+ \varepsilon +{ \eta}_{1i}, $$where (Aw1)i denote the component i of Aw1. Using this join with (38) we have
$$ 0={{\alpha}_{1i}^{*}}((\mathbf{A}\mathbf{w}_{1})_{i}+b_{1}- y_{i}- \varepsilon -{\eta}_{1i}^{*})=-{\alpha}_{1i}^{*}(2\varepsilon +{\eta}_{1i}+{\eta}_{1i}^{*}). $$Since \(\boldsymbol {\eta }_{1},\boldsymbol {\eta }_{1}^{*}\ge 0 \) and ε > 0, we obtain that necessarily \({\alpha }_{1i}^{*}=0.\) The same argument can be used to prove that \({\alpha }_{1i}^{*}>0\) implies that α1i = 0, and similar arguments apply to the case \({\alpha }_{2i}\cdot {\alpha }_{2i}^{*}=0.\)
- (c)
We note first that \((\mathbf {A}\mathbf {w}_{1})_{j}=\mathbf {w}_{1}^{\top } \mathbf {x}_{j}. \) Assume that \({ \alpha }_{1j}\in (0,\hat {c}_{1}),\) from (30) we have that γ1j > 0 and then, by (40) η1j must be equal to zero. Therefore, from (38) we obtain
$$ b_{1}=y_{j}-(\mathbf{A}\mathbf{w}_{1})_{j}-\varepsilon=y_{j}-\mathbf{w}_{1}^{\top} \mathbf{x}_{j}-\varepsilon. $$We can now proceed analogously to the case \({ \alpha }_{1j}^{*}\in (0,\hat {c}_{1})).\)
- (d)
The same arguments developed above can be used in this case.
- (e)
Assume β1j ∈ (0, c1) for some j ∈ 1, …m1, we have from (32) that \({ \beta }_{1j}^{*}>0\) and then, it follows from (40) that ξ1j = 0. Finally, from (39) we obtain directly that
$$ y_{j}-\mathbf{w}_{1}^{\top} \mathbf{x}_{j}-b_{1}+\varepsilon_{1}=0. $$
□
Rights and permissions
About this article
Cite this article
Carrasco, M., López, J. & Maldonado, S. Epsilon-nonparallel support vector regression. Appl Intell 49, 4223–4236 (2019). https://doi.org/10.1007/s10489-019-01498-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-019-01498-1