Multi-Objective Model Selection Via Racing

Keywords

Discrete Holm's step-down procedure; model selection; multi-objective optimization; racing algorithm (RA); sign test (ST)

Abstract

Model selection is a core aspect in machine learning and is, occasionally, multi-objective in nature. For instance, hyper-parameter selection in a multi-task learning context is of multi-objective nature, since all the tasks' objectives must be optimized simultaneously. In this paper, a novel multi-objective racing algorithm (RA), namely S-Race, is put forward. Given an ensemble of models, our task is to reliably identify Pareto optimal models evaluated against multiple objectives, while minimizing the total computational cost. As a RA, S-Race attempts to eliminate non-promising models with confidence as early as possible, so as to concentrate computational resources on promising ones. Simultaneously, it addresses the problem of multi-objective model selection (MOMS) in the sense of Pareto optimality. In S-Race, the nonparametric sign test is utilized for pair-wise dominance relationship identification. Moreover, a discrete Holm's step-down procedure is adopted to control the family-wise error rate of the set of hypotheses made simultaneously. The significance level assigned to each family is adjusted adaptively during the race. In order to illustrate its merits, S-Race is applied on three MOMS problems: 1) selecting support vector machines for classification; 2) tuning the parameters of artificial bee colony algorithms for numerical optimization; and 3) constructing optimal hybrid recommendation systems for movie recommendation. The experimental results confirm that S-Race is an efficient and effective MOMS algorithm compared to a brute-force approach.

Publication Date

8-1-2016

Publication Title

IEEE Transactions on Cybernetics

Volume

46

Issue

8

Number of Pages

1863-1876

Document Type

Article

Personal Identifier

scopus

DOI Link

https://doi.org/10.1109/TCYB.2015.2456187

Socpus ID

84939168171 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/84939168171

This document is currently not available here.

Share

COinS