AIWolfDial SharedTask 2019 : Call for Shared Task Participation: AI Werewolf and Dialog System (AIWolfDial2019)
Call For Papers
Task registration and preliminary run submission extended to Aug 23!
2nd Call for Shared Task Participation
AI Werewolf and Dialog System (AIWolfDial2019)
As part of the 1st International Workshop of AI Werewolf and Dialog System
Collocated with INLG 2019 conference, October 29, 2019, Tokyo, Japan
( Aims )
"Are You a Werewolf?", or "Mafia" (hereafter "werewolf game"), is a communication game conducted solely through discussion. Players must exert their cognitive faculties fully in order to win. In the game, players must hide information, in contrast to perfect information games such as chess or Reversi. Each player acquires secret information from other players’ conversations and behavior and acts by hiding information to accomplish their objectives. Players are required persuasion for earning confidence, and speculation for detecting fabrications.
We employ this werewolf game as a novel way of evaluations for dialog systems. While studies of dialog systems are very hot topics recently, they are still insufficient to make natural conversations with consistent context, or with complex sentences. One of the fundamental issues is a lack of an appropriate evaluation.
Because the werewolf game forces players to deceive, persuade, and detect lies, neither inconsistent nor vague response are evaluated as “unnatural”, losing in the game. Our werewolf game competition and evaluation could be a new interesting evaluation criteria for dialog systems, but also for imperfect information game theories.
In addition, the werewolf game allows any conversation, so the game includes both task-oriented and non-task-oriented conversations. This aspect would provide a handy intermediate goal rather than to create a general dialog system from scratch.
( Important dates )
[EXTENDED] Shared task preliminary run and registration due: August 23 (AoE), 2019
Shared task formal run: end of August
Shared task evaluation result feedback: beginning of September
Submissions due: Sep 10, 2019
Notification of acceptance: October 1, 2019
Camera-ready papers due: October 15, 2019
Workshop: October 29, 2019
( Shared task )
* Background *
We have been annually holding the AI werewolf contests under the AI Werewolf project. The AI werewolf contest has two divisions, the protocol division and the natural language division. The protocol division asks participants to implement an AI werewolf player agent that communicates in a middle language called the AI werewolf protocol. The natural language division asks participants to implement an AI werewolf agent that communicates in natural language. We follow the previous configurations of the natural language division in our AIWolfDial2019 shared task.
* The Werewolf Game in this Shared Task *
Agents of the AIWolfDial2019 shared task will play the werewolf game of five players, including roles of a seer, a werewolf, a possessed, and two villagers. Players do not know other players' roles. All players, other than the werewolf, are humans.
A game consists of a couple of days, continuing until the human team or the werewolf team survives. A werewolf can specify and attack another player in the end of the day; the attached player will be eliminated from the game. All of players are required to vote to another player, and a player voted most will be eliminated from the game. When humans survive, the villager team wins. When a werewolf survives, the werewolf team wins. A possessed is a human but belongs to the werewolf team. A seer can specify another player in the end of the day, then either human or werewolf is notified.
* Agent Technical Specifications *
A shared task participant of AIWolfDial2019 is required to implement an AI werewolf agent that communicate either in English or Japanese. Agents of the Japanese language are required to make an English version, at least by using machine translation internally.
AIWolf Agent APIs
A shared task participant of AIWolfDial2019 is required to implement an AI werewolf agent that connects to our AIWolf server via network. Therefore, there is no limitation on the client side hardware/software other than the network I/O specifications.
We recommend to use our sample agent code with runtime environment (java), distributed from our Github repository prepared for this shared task:
Please clone the repository to your Eclipse environment and run the org.aiwolf.ui.bin.AutoStarter class with specifying AutoStarter.ini as its program argument. This configuration runs the same five random talk agents. We provide our test server to connect and test your agent as same as the preliminary/formal run.
- Day 0 has greetings only.
- The end of Day 0 has an inspection by a seer, and the game starts from Day 1.
- After Day 1, the end of the days have votes by all players and an attack by a werewolf. Vote, attack, inspection are made via specific APIs (network communications).
- A day consists of a couple turns, where all of agents can make a talk for each turn, receiving talks of previous turns.
- An agent should make a talk within a specified periods after a talk request is sent.
- During days, Agents can communicate anything in natural language. A talk should consists of normal letters and punctuations only. An agent returns "Skip" when nothing to talk, returns "Over" if nothing to talk anymore in that day.
- Use Agent[0x] (e.g. Agent, x is 1-5) to mention other agents．
- An anchor e.g. "))Agent[0x]" could be inserted at the beginning of a talk to refer to another agent, to whom your agent with to talk with. That agent is assumed to respond something to your agent by using an anchor.
* From Registration to Formal Run *
A team should send a mail to aiwolf at kanolab.net (replace at by @) to register the shared task, describing your team name, a contact e-mail address, names and affiliations of your members (please mark a contact person when a team consists of multiple members), communication language (English and/or Japanese) of your agent. There is no fee required to register/participate the shared task.
Testing Your Agent System Beforehand
A shared task participant of AIWolfDial2019 is required to implement an AI werewolf agent that connects to our AIWolf server at the specified timing. We will provide an AIWolf server running, where participants can try connecting with dummy agents to check their system behavior. Participants are required to check their systems certainly work before the shared task run.
Participants should run your five agents connected to our server, running 50 games. Then submit your game logs to the organisers via e-mail. If there are too many participants to run the formal run, organisers might select formal run teams depending on these logs. In either case, these logs will be used in the final evaluations.
A participant team is required to connect an agent to our server at a specified timing to play games with other participants. If your team cannot remotely attend the formal run, you should prepare your system where the organisers can somehow trigger your remote system to connect our server and run, e.g. by via a web browser.
Participants should submit a system design description document to the organizers. This document and logs of the games might be used for research purpose and included and published in our overview paper without any further permission. Participants are encouraged to submit a paper to the workshop.
Games will be between the same agents, different agents, and/or human players. Reviewers will perform subjective evaluations on the game logs, using following criteria:
A Natural utterance expressions
B Contextually natural conversation
C Coherent (not contradictory) conversation
D Coherent game actions (vote, attack, divine) with conversation contents
E Diverse utterance expressions, including coherent characterization
Please not that vague utterances, that can be used regardless of contexts, are not always natural in the werewolf game.
( Committee )
E-mail to aiwolf at kanolab.net (replace at by @)
Yoshinobu Kano, Shizuoka University, Japan
Michimasa Inaba, The University of Electro-Communications, Japan
Fujio Toriumi, The University of Tokyo, Japan
Hirotaka Osawa, University of Tsukuba, Japan
Daisuke Katagami, Tokyo Polytechnic University, Japan
Takashi Otsuki, Yamagata University, Japan
Program committee members
Claus Aranha, Tsukuba University, Japan
Hitoshi Matsubara, Future University Hakodate, Japan
( Workshop )
Please see our workshop website for details of the AIWolfDial workshop, including paper submissions: