posted by organizer: bnaderi || 513 views || tracked by 1 users: [display]

Crowd@QoMEX 2021 : Crowd-sourced and Remote User Studies for Quality of Experience and Usability Research


When Jun 14, 2021 - Jun 17, 2021
Where virtual
Submission Deadline Feb 16, 2021
Notification Due Mar 28, 2021
Categories    crowdsourcing   quality of experience   user experience   remote testing

Call For Papers

Special session on Crowd-sourced and Remote User Studies for Quality of Experience and Usability Research at QoMEX 2021 | 14 - 17 June 2021 | Virtual

13th International Conference on Quality of Multimedia Experiences

Important Dates:
- 20th January 2021: Grant application (optional, see below)
- 21st January 2021: Grant acceptance notice
- 16th February 2021: Full paper submission
- 28th March 2021: Full paper notification

PAPERS SUBMISSION : Please check out the QoMEX 2021 timeline and guidance, as the papers should be submitted following the conference instructions.

Publication: IEEE

Grant: $ 200 financial support provided by for paper ideas who want use the for their experiment and their grant application is accepted. See below for details.

Dear Colleagues:
We organize a special session on Crowd-sourced and Remote User Studies for Quality of Experience and Usability Research at QoMEX 2021.
Laboratory studies are an established and essential tool for Quality of Experience (QoE) and User Experience (UX) research. However, they required well-equipped test rooms and personnel for the supervision of the test participants. Therefore, they are often cost- and time-intensive. Further, the number of test candidates is often limited due to the sparse laboratory space and need for the test taskers' physical presence in the test environment. In the past months, the pandemic situation made it even more challenging to conduct laboratory studies by increasing the organizational overhead and limiting potential participants. Two possibilities to overcome the current situation are crowdsourcing and remote user studies.
In this context, the special session aims to foster contributions following the traditional way of optimizing and designing crowdsourced subjective studies for Quality of Experience and User Experience research and additionally wants to raise awareness and foster research in a new research direction, the Quality of Experience and User Experience of crowdsourcing workers. The special session also seeks to encourage researchers to exchange their experiences in remote user studies, with non-anonymous test takers, how best practices from crowdsourcing studies can be applied in this context, and discuss which new challenges arise.

The topics of interest include but are not limited to the following:
• Crowdsourcing for subjective studies
• Novel applications
• Limitations of current crowdsourcing systems
• Quality control mechanism and reliability metrics
• Large scale crowdsourcing studies and diversity of participants
• Reproducibility of results
• Reproducibility and cross platform studies
• Assessment of hidden influence factors / Impact of hidden influence factors
• Bias estimation and bias reduction
• Automation and workflows
• Standardization of crowdsourcing test methods
• Usability and User Experience of crowdsourcing tasks
• Optimization of task interfaces and task workflows
• Relation to result quality and worker motivation
• Enhancing workers’ UX (e.g., by means of gamification of tasks)
• Quality of complex crowdsourcing workflows (e.g. combination of AI and Crowds)
• Interconnection of crowdsourcing and lab-based tests,
• Studies comparing results from lab and crowdsourcing
• Adaptations of established lab-test standards to the crowdsourcing environment
• Remote user studies
• Supervised remote user studies
• Remote studies with non-anonymous users
• (Crowdsourcing) best practices for remote user studies

Authors are kindly invited to submit their full papers by the deadline. All submissions are handled through the QoMEX2021 submission process:
For any query, Please write mail to babak.naderi@tu-berlin and

Sincerely Yours,

Matthias Hirth, Institute of Media Technology, TU Ilmenau,
Babak Naderi, Quality and Usability Lab, Technical University of Berlin

Grant details:

In line to support research on the QoE using crowdsourcing, offers $200 grants for selected ideas who want to conduct an experiment on their platform and submit a paper to this special session. The applicants have to send a research abstract and a rough cost plan how the grand will be spend to the to the session organizers. All applications will be forwarded to where a company representative will make the final funding decision. Note, the funds cannot be withdrawn from Microworkers and can only be used for research purposes.
Please send your 1)research abstract, 2) cost plan by20th January (AOE) to and The selected candidates will be announced on 21th of January.

Related Resources

DigEx 2021   3rd International Workshop on Transforming the Digital Customer Experience
BlockArch 2021   Second Workshop on Blockchain-Based Software Architectures (BlockArch 2021)
FUSION 2021   National Symposium on Human-Computer Interaction
IUI 2021   ACM Conference on Intelligent User Interfaces
FUSION 2021   National Symposium on Human-Computer Interaction
ACM UMAP 2021   29th Conference on User Modeling, Adaptation and Personalization
FUSION 2021   National Symposium on Human-Computer Interaction
QUATIC 2021   International Conference on the Quality of Information and Communications Technology
SIGPRO 2021   International Conference on Data Science and Cloud Computing
DLRS 2021   Call for Papers: Topical Issue on Deep Learning for Recommender Systems