ICML ML Evaluation 2009 : ICML 2009 Workshop on Evaluation Methods for Machine Learning IV
Call For Papers
Call For Submissions to ICML 2009 Workshop
The Fourth Workshop on Evaluation Methods for Machine Learning
June 18, 2009 Montreal Canada.
William Klement, Chris Drummond, Nathalie Japkowicz, and Sofus Macskassy.
The fourth in a series, this workshop intends to continue the debate within the
machine learning community into how we evaluate new algorithms. We aim to discuss
what properties of an algorithm need to be evaluated (e.g., accuracy,
comprehensibility, conciseness); to solicit views and suggestions for other
approaches than those currently used; to investigate alternate methods that could
In the course of three previous workshops, the debate has evolved to focus around
the following issues which have captured the interest of the community:
* the role of experiments in evaluation
* the use of one, community wide, evaluation measure (e.g., Accuracy, AUC,
* the relevance of statistical tests to evaluation
* the effectiveness of the UCI data sets for evaluation
* the need for sharing and characterizing benchmark data sets in general
* how to promote the views of this workshop to the rest of the community
The 2008 ICML workshop concluded with agreement that we, as a scientific community,
should substantially change how evaluation is performed in machine learning. We,
however, disagreed on the direction that this change should take. As a continuation
of the same theme, we aim to solicit views, intuitions and visions of alternatives
to change existing evaluation methods. We hope to make progress but still carry
forward the good methods and experiences we already have acquired.
We invite position and technical papers concerning the following issues:
* advantages of existing evaluation methods
* critiques of current evaluation practices
* intuitive and creative alternatives
* performance evaluation issues of concern to the community
* future directions and evolutions of evaluation methods in machine
This list certainly does not capture all the issues worthy of discussion nor the
possible positions. We expect, and very much encourage, position papers raising
other issues that members of the machine learning community think are important.
The day-long workshop consists of:
Invited talks: we plan to have several invited speakers. Some, from
outside of our research community, will be able to criticize our accepted
practices from an external point of view. Some, from inside our community,
will discuss how we could improve on our current practices.
Panel Discussion: our invited speakers will be asked to engage each other on
the various issues surrounding the problem of evaluation in Machine Learning
at the end of the workshop. The audience will be strongly encouraged to
participate in the discussion.
Presentations: The papers accepted to the workshop will be presented throughout
the day between the various invited talks. Papers will be grouped by theme, in
order to facilitate discussion at the end of each session.
Workshop attendance is open to the public and is estimated at 20-25 attendees.
Priority will be given to those active participants in the workshop (paper
authors or speakers).
Authors are invited to submit papers on the issues listed above or other
related positions. Technical or position reports will be considered for this
workshop. To promote a lively event, with plenty of discussion, the organizers
are very interested in papers taking strong positions on the above topics.
Workshop papers should be at least 1 page long and should not exceed 4 pages using
the ICML'09 Style. Submissions should be made electronically in PDF or Postscript
format and should be sent (no later than April 8, 2009 -- EXTENDED to April 16, 2009) by email to:
William Klement, email:
klement -at- site -dot- uottawa -dot- ca
Each submissions will be reviewed by two members of the organizing committee.
A notification of acceptance will be sent by: April 30, 2009. Authors will be
asked to submit a final version of their submission to be posted on the workshop
April 8, 2009 (Extended to April 16, 2008) Submission due date
April 30, 2009 Author notification Date
June 1, 2009 Final version ready
June 14-18, 2009 ICML 2009 Conference
June 18, 2009 Workshop Date
William Klement (main contact)
SITE, University of Ottawa
800 King Edward Ave.
P.O. Box 450, Stn A
Ottawa, ON K1N 6N5, Canada.
Telephone: (613) 562-5800 ext. 6699
Fax: (613) 562-5664
NRC Institute for Information Technology, Canada
Email: chris.drummond -at- nrc-cnrc -dot- gc -dot- ca
School of Information Technology and Engineering
University of Ottawa, Canada
Email: nat -at- site -dot- uottawa -dot- ca
Sofus A. Macskassy
United States of America
Email: sofmac -at- fetch -dot- com