Title: Look before you leap: Some insights into learner evaluation with cross-validation
Authors: Vanwinckelen, Gitte
Blockeel, Hendrik
Issue Date: 15-Sep-2014
Host Document: JMLR: Workshop and Conference Proceedings pages:1-17
Conference: ECML/PKDD Workshop on Statistically Sound Data Mining location:Nancy, France date:15 September 2014
Abstract: Machine learning is largely an experimental science, of which the evaluation of predictive
models is an important aspect. These days, cross-validation is the most widely used method
for this task. There are, however, a number of important points that should be taken into
account when using this methodology. First, one should clearly state what they are trying to
estimate. Namely, a distinction should be made between the evaluation of a model learned
on a single dataset, and that of a learner trained on a random sample from a given data
population. Each of these two questions requires a different statistical approach and should
not be confused with each other. While this has been noted before, the literature on this
topic is generally not very accessible. This paper tries to give an understandable overview
of the statistical aspects of these two evaluation tasks. We also pose that because of the
often limited availability of data, and the difficulty of selecting an appropriate statistical
test, it is in some cases perhaps better to abstain from statistical testing, and instead focus
on an interpretation of the immediate results.
Publication status: published
KU Leuven publication type: IC
Appears in Collections:Informatics Section

Files in This Item:
File Description Status SizeFormat
gittevanwinckelen.pdfLook before you leap: Some insights into learner evaluation with cross-validation Accepted 267KbAdobe PDFView/Open


All items in Lirias are protected by copyright, with all rights reserved.