posted by user: grupocole || 3419 views || tracked by 8 users: [display]

FIRE 2009 : SIGIR 2009 Workshop on the Future of IR Evaluation

FacebookTwitterLinkedInGoogle

Link: http://staff.science.uva.nl/~kamps/ireval/
 
When Jul 23, 2009 - Jul 23, 2009
Where Boston, MA, USA
Submission Deadline May 18, 2009
Notification Due Jun 8, 2009
Final Version Due Jun 15, 2009
Categories    information retrieval
 

Call For Papers

----
SIGIR 2009 Workshop on the Future of IR Evaluation July 23, Boston
http://staff.science.uva.nl/~kamps/ireval/

Call for Papers

Evaluation is at the core of information retrieval: virtually all progress
owes directly or indirectly to test collections built within the so-called
Cranfield paradigm. However, in recent years, IR researchers are routinely
pursuing tasks outside the traditional paradigm, by taking a broader view on
tasks, users, and context.
There is a fast moving evolution in content from traditional static text to
diverse forms of dynamic, collaborative, and multilingual information
sources. Also industry is embracing "operational"
evaluation based on the analysis of endless streams of queries and clicks.

We invite the submission of papers that think outside the box:

- Are you working on an interesting new retrieval task or aspect? Or
on its broader task or user context? Or on a complete system with
novel interface? Or on interactive/adaptive search? Or ...?
Please explain why this is of interest, and what would be an
appropriate way of evaluating.

- Do you feel that the current evaluation tools fail to do justice to
your research? Is there a crucial aspect missing? Or are you
interested in specific, rare, phenomena that have little impact on
the average scores? Or ...? Please explain why this is of interest,
and what would be an appropriate way of evaluating.

- Do you have concrete ideas how to evaluate such a novel IR task? Or
ideas for new types of experimental or operational evaluation? Or
new measures or ways of re-using existing data? Or ...? Please
explain why this is of interest, and what would be an appropriate
way of evaluating.

The workshop brings together all stake-holders ranging from those with novel
evaluation needs, such as a PhD candidate pursuing a new IR-related problem,
to senior IR evaluation experts. Desired outcomes are insight into how to
make IR evaluation more "realistic," and at least one concrete idea for a
retrieval track or task (at CLEF, INEX, NTCIR, TREC) that would not have
happened otherwise.

Help us shape the future of IR evaluation!

- Submit a short 2-page poster or position paper explaining your key
wishes or key points,

- and take actively part in the discussion at the Workshop.

The deadline is Monday May 18, 2009, further submission details are on
http://staff.science.uva.nl/~kamps/ireval/


Shlomo Geva, INEX & QUT, Australia
Jaap Kamps, INEX & University of Amsterdam, The Netherlands
Carol Peters, CLEF & ISTI-CNR, Italy
Tetsuya Sakai, NTCIR & Microsoft Research Asia, China
Andrew Trotman, INEX & University of Otago, New Zealand
Ellen Voorhees, TREC/TAC & NIST, USA



Related Resources

SIGIR 2025   The 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
LSIJ 2025   Life Sciences: an International Journal
IEEE FMLDS 2025   2025 IEEE International Conference on Future Machine Learning and Data Science (FMLDS2025)
TPDL 2025   29th International Conference on Theory and Practice of Digital Libraries
ICFCC--EI 2025   2025 The 17th International Conference on Future Computer and Communication (ICFCC 2025)
EI/Scopus-CMNM 2025   2025 2025 2nd International Conference on Modeling, Natural Language Processing, and Machine Learning-EI/Scopus
BS LAB 2025   9th Business Systems Laboratory International Symposium TECHNOLOGY AND SOCIETY: Boon or Bane?
KDIR 2025   17th International Conference on Knowledge Discovery and Information Retrieval
EVALSUP & CIMQUSEF 2025   The 7th International Conference on Evaluation in Higher Education and the 22th International Conference on the Quality of Education
FGCS 2025   Special Issue on “Advanced Situational Awareness and Resilience for Hyperconnected Industrial Scenarios” in Future Generation Computer Systems (FGCS), Elsevier