Pareto-Optimal Model Selection Via Sprint-Race

Keywords

Model selection (MS); multi-objective optimization; racing algorithm; sequential probability ratio test (SPRT)

Abstract

In machine learning, the notion of multi-objective model selection (MOMS) refers to the problem of identifying the set of Pareto-optimal models that optimize by compromising more than one predefined objectives simultaneously. This paper introduces SPRINT-Race, the first multi-objective racing algorithm in a fixed-confidence setting, which is based on the sequential probability ratio with indifference zone test. SPRINT-Race addresses the problem of MOMS with multiple stochastic optimization objectives in the proper Pareto-optimality sense. In SPRINT-Race, a pairwise dominance or non-dominance relationship is statistically inferred via a non-parametric, ternary-decision, dual-sequential probability ratio test. The overall probability of falsely eliminating any Pareto-optimal models or mistakenly returning any clearly dominated models is strictly controlled by a sequential Holm's step-down family-wise error rate control method. As a fixed-confidence model selection algorithm, the objective of SPRINT-Race is to minimize the computational effort required to achieve a prescribed confidence level about the quality of the returned models. The performance of SPRINT-Race is first examined via an artificially constructed MOMS problem with known ground truth. Subsequently, SPRINT-Race is applied on two real-world applications: 1) hybrid recommender system design and 2) multi-criteria stock selection. The experimental results verify that SPRINT-Race is an effective and efficient tool for such MOMS problems.

Publication Date

2-1-2018

Publication Title

IEEE Transactions on Cybernetics

Volume

48

Issue

2

Number of Pages

596-610

Document Type

Article

Personal Identifier

scopus

DOI Link

https://doi.org/10.1109/TCYB.2017.2647821

Socpus ID

85011653983 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/85011653983

This document is currently not available here.

Share

COinS