Generalized error path algorithm | 数据学习(DataLearner)

Highlights：

• we first show that the solution paths produced by various algorithms have the property of piecewise linearity.

• We point out model function builds the bridge between solution path and error path, and show that the piecewise linearity of solution path leads to the piecewise linearity of model function.

• Based on the piecewise linearity of model function, we prove that a large class of error (or loss) functions are piecewise constant, linear, or quadratic w.r.t. the regularization parameter. Finally, we propose our GEP for the generalized error (or loss) functions and solution path algorithms, which guarantees to find the models with the minimum CV error.

• The experimental results on a variety of datasets not only confirm our theoretical findings, but also show that the best model with our GEP has better generalization error on the test data, compared to the grid search, manual search, and random search.

摘要

•we first show that the solution paths produced by various algorithms have the property of piecewise linearity.•We point out model function builds the bridge between solution path and error path, and show that the piecewise linearity of solution path leads to the piecewise linearity of model function.•Based on the piecewise linearity of model function, we prove that a large class of error (or loss) functions are piecewise constant, linear, or quadratic w.r.t. the regularization parameter. Finally, we propose our GEP for the generalized error (or loss) functions and solution path algorithms, which guarantees to find the models with the minimum CV error.•The experimental results on a variety of datasets not only confirm our theoretical findings, but also show that the best model with our GEP has better generalization error on the test data, compared to the grid search, manual search, and random search.