南京航空航天大学：《模式识别》课程教学资源（PPT讲稿）Model Selection for SVM & Our intent works

Model selection for svm Our intent works Songcan Chen Feb.8,2012

Model Selection for SVM & Our intent works Songcan Chen Feb. 8, 2012

Outline Model selection for svm Our intent works

Outline • Model Selection for SVM • Our intent works

Model selection for svm Introduction to 2 works

Model Selection for SVM • Introduction to 2 works

Introduction to 2 works 1. Model selection for primal SVM [MBB11, MLJ111 2. Selection of Hypothesis Space Selecting the Hypothesis Space for Improving the Generalization ability of Support Vector Machines [AGOR11, IJCNN20111 The Impact of Unlabeled patterns in Rademacher Complexity Theory for Kernel Classifiers [AGOR11, NIPS20111

Introduction to 2 works 1. Model selection for primal SVM [MBB11, MLJ11] 2. Selection of Hypothesis Space • Selecting the Hypothesis Space for Improving the Generalization Ability of Support Vector Machines [AGOR11,IJCNN2011] • The Impact of Unlabeled Patterns in Rademacher Complexity Theory for Kernel Classifiers [AGOR11,NIPS2011]

1st work Model selection for primal sV [MBB11, MLJ111 IMBBllGregory Moore Charles bergeron Kristin P. Bennett Machine Learning(2011)85: 175-208

1 st work • Model selection for primal SVM [MBB11, MLJ11] [MBB11] Gregory Moore · Charles Bergeron · Kristin P. Bennett, Machine Learning (2011) 85:175–208

Outline Primal svm Model selection 1)Bilevel Program for Cv 2)TWo optimization Methods Implicit EXplicit methods 3)Experiments 4)Conclusions

Outline • Primal SVM • Model selection 1) Bilevel Program for CV 2) Two optimization Methods: Impilicit & Explicit methods 3) Experiments 4) Conclusions

Primal svm Advantages 1) simple to implement, theoretically sound, and easy to customize to different tasks such as classification, regression, ranking and so forth 2)very fast, linear in the number of samples · Difficulty model selection

Primal SVM • Advantages: 1) simple to implement, theoretically sound, and easy to customize to different tasks such as classification, regression, ranking and so forth. 2) very fast, linear in the number of samples • Difficulty model selection

Model selection An often-adopted approach Cross-validation(Cv over a grid Advantage simple and almost universal Weakness high computation exponential in the number of hyperparameters and the number of grid points for each hyperparameter

Model selection An often-adopted approach: Cross-validation (CV) over a grid Advantage: simple and almost universal! Weakness: high computation exponential in the number of hyperparameters and the number of grid points for each hyperparameter

Motivation CV is naturally and precisely formulated as a bilevel program ( BP)shown as follows LEADER outer-level min Y val Bilevel CV Problem model hyperparameters (BCP) weights FOLLOWER inner-level minw Ctm(w, Y)

Motivation • CV is naturally and precisely formulated as a bilevel program (BP) shown as follows. Bilevel CV Problem (BCP)

Bilevel CV Problem(BCP)( BCP for a single validation and training split The outer-level leader problem selects the nyperparameters, to perform well on a validation set The follower problem trains an optimal inner-level model for the given hyperparameters, and returns a weight vector for validation

Bilevel CV Problem (BCP) (1) BCP for a single validation and training split: • The outer-level leader problem selects the hyperparameters, γ, to perform well on a validation set. • The follower problem trains an optimal inner-level model for the given hyperparameters, and returns a weight vector w for validation