NAACL-HLT 2010 Workshop on Amazon Mechanical Turk
June 5 or 6, Los Angeles, CA

Workshop Website: http://sites.google.com/site/amtworkshop2010/

Submission Deadline: March 1, 2010
Shared Task Proposal Deadline: January 31, 2010

Creating Speech and Text Language Data With Amazon's Mechanical Turk

Amazon's Mechanical Turk (AMT) is an online labor market that allows
you to pay people small sums of money to do "Human Intelligence Tasks"
or HITs. Tasks include labeling images, listening to short pieces of
audio, researching topics on the internet, and scrubbing database
records.

A number of recent papers have evaluated the effectiveness of using
Mechanical Turk to create annotated data for natural language
processing applications. The availability of Mechanical Turk for
creating annotated speech and text language data can fundamentally
change human language technology tasks. Open questions include:
- How can we ensure high quality annotations?
- What tools are available for obtaining complex annotations?
- What types of annotations and evaluations are possible when the cost
is dramatically reduced?

This work will explore uses of AMT in several ways:
- Shared task: What can you do with $100 and AMT? Participants will be
given a budget to spend on AMT and submit papers describing the
results of their experience.
- General papers: These papers will explore general issues with using
Mechanical Turk for language processing research.

Important Dates
Jan 31, 2010 Shared Task Proposal Deadline (for teams wishing to
be considered for funding)
Mar 1, 2010 Paper submission deadline
Mar 30, 2010 Notification of acceptance
Apr 12, 2010 Camera-ready papers due
Jun 5-6, 2010 Workshop at NAACL in Los Angeles, CA (exact date TBA)

Workshop Website
http://sites.google.com/site/amtworkshop2010/

Organizing Committee
Chris Callison-Burch (Johns Hopkins University)
Mark Dredze (Johns Hopkins University)

Program Committee
Jordan Boyd-Graber (UMD)
Michael Bloodgood (HLTCOE)
Bob Carpenter (Alias-i)
James Dennis (J2 Labs)
Maxine Eskenazi (CMU)
Nikesh Garera (Kosmix)
Jim Glass (MIT)
Alex Gruenstein (Google)
Janna Hamaker (Amazon)
Benjamin Lambert (CMU)
Alexandre Klementiev (JHU)
Ian McGraw (MIT)
Scott Novotney (JHU)
Brendan O'Connor (Dolores Labs)
Massimo Poesio (University of Essex)
Joseph Reisinger (UT Austin)
Ted Sandler (Amazon)
Stephanie Seneff (MIT)
Rion Snow (Stanford / Twitter)

We thank Amazon Mechanical Turk for supporting the shared task.