posted by user: wsun || 1877 views || tracked by 4 users: [display]

MAGG 2012 : 2012 AAAI Fall Symposium on Machine Aggregation of Human Judgment


When Nov 2, 2012 - Nov 4, 2012
Where Arlington, Virginia
Submission Deadline May 25, 2012
Notification Due Jun 22, 2012
Final Version Due Sep 7, 2012
Categories    machine learning   cloudsourcing   information fusion   collective intelligence

Call For Papers

The AAAI 2012 Fall Symposium on Machine Aggregation of Human Judgment focuses on combining human and machine inference. For unique events and data-poor problems, there is no substitute for human judgment. Even for data-rich problems, human input is needed to account for contextual factors. For example, textual analysis is data rich, but context and semantics often make automated parsing unusable. However, humans are notorious for underestimating the uncertainty in their forecasts and even the most expert judgments exhibit well-known cognitive biases. The challenge is therefore to aggregate expert judgment such that it compensates for the human deficiencies.

There are fundamental theoretical reasons to expect aggregated estimates to out-perform individual forecasts. These theoretical results are borne out by a robust empirical literature demonstrating the superiority of opinion pools and prediction markets over individual forecasts, and of ensemble forecasts over those of top models. While weighted forecasts are theoretically optimal, among human experts unweighted forecasts have been hard to beat.

This symposium focuses on methods with the potential to come closer to the theoretical optimum. While a number of methods have shown promise individually, there is potential for significant advancement from combining them into structured, efficient, repeatable elicitation and aggregation protocols. Benefits of improved aggregation methods include substantial increases in the quality and reliability of expert judgements, removing misunderstanding, illuminating context dependence of forecasts, and reducing overconfidence and motivational bias in forecasts. On the other hand, there’s some skepticism that statistical models can outperform experts most of the time. Machine reasoning lacks the context to know when the models no longer apply, or in cases like natural language, simply lack sufficient context to be reliable in open-world or novel problems. This symposium considers powerful hybrid techniques using humans to help aggregate computer models.

A broad range of researchers in the AI community and other application fields such as econometrics, sociology, political science, and intelligence analysis will find this symposium interesting and useful. Bringing these disciplines together to the venue also greatly facilitates the research endeavors.

The symposium will accept a number of regular papers (6-8 pages), and short papers/extended abstracts (2 pages). In addition to oral presentations, we intend to provide several poster sessions for more interactions. Further, invited talks by leading researchers in the fields and/or domain experts will be arranged. We will also reserve substantial time for questions and discussions after talks.

Topics include but not limited to the following
• Reasoning under uncertainty
• Ensembles & aggregation
• Information fusion
• Crowdsourcing techniques & applications
• Information elicitation & presentation
• Performance evaluation: scalability and accuracy
• Prediction markets
• Collective intelligence
Submission should be done through EasyChair submission site. Authors, who do not have accounts on EasyChair, will be directed to create a new account before they can make submission.

Important Dates
• Friday, May 25, 2012, 11:59pm Eastern Time – Papers/abstracts due.
• Friday, June 22, 2012 – Author notification of acceptance/rejection.
• Friday, September 07, 2012 – Camera-ready paper due.
• Friday November 02 – Sunday November 04, 2012 – Symposium at Arlington, Virginia.

Organizing Committee
Kathryn B. Laskey, Wei Sun (George Mason University), John Irvine (Draper Laboratory), Dirk B. Warnaar (Applied Research Associates, Inc.), H. Van Dyke Parunak (Jacobs Technology Inc.)

Supplementary Website
For more information about the symposium, and for submission guidelines and links, please visit the supplementary website (

Related Resources

ICDM 2020   20th IEEE International Conference on Data Mining
AAAI 2020   The Thirty-Fourth AAAI Conference on Artificial Intelligence
AAAI-MAKE 2020   AAAI 2020 Spring Symposium on Combining Machine Learning and Knowledge Engineering in Practice
MNLP 2020   4th IEEE Conference on Machine Learning and Natural Language Processing
IEEE-CVIV 2020   2020 2nd International Conference on Advances in Computer Vision, Image and Virtualization (CVIV 2020)
SI-FDS-MLJ 2021   CFP: Special Issue on Foundations of Data Science - Machine Learning Journal
FAIML 2020-Ei Compendex & Scopus 2020   2020 2nd International Conference on Frontiers of Artificial Intelligence and Machine Learning (FAIML 2020)
ACM--ICMLC--Ei and Scopus 2020   ACM--2020 12th International Conference on Machine Learning and Computing (ICMLC 2020)--SCOPUS, Ei Compendex
WSPML 2020   2020 2nd International Workshop on Signal Processing and Machine Learning (WSPML 2020)
ICANN 2020   29th International Conference on Artificial Neural Networks