posted by organizer: bnaderi || 1076 views || tracked by 1 users: [display]

Crowd@QoMEX 2021 : Crowd-sourced and Remote User Studies for Quality of Experience and Usability Research


When Jun 14, 2021 - Jun 17, 2021
Where virtual
Submission Deadline Feb 16, 2021
Notification Due Mar 28, 2021
Categories    crowdsourcing   quality of experience   user experience   remote testing

Call For Papers

Special session on Crowd-sourced and Remote User Studies for Quality of Experience and Usability Research at QoMEX 2021 | 14 - 17 June 2021 | Virtual

13th International Conference on Quality of Multimedia Experiences

Important Dates:
- 20th January 2021: Grant application (optional, see below)
- 21st January 2021: Grant acceptance notice
- 16th February 2021: Full paper submission
- 28th March 2021: Full paper notification

PAPERS SUBMISSION : Please check out the QoMEX 2021 timeline and guidance, as the papers should be submitted following the conference instructions.

Publication: IEEE

Grant: $ 200 financial support provided by for paper ideas who want use the for their experiment and their grant application is accepted. See below for details.

Dear Colleagues:
We organize a special session on Crowd-sourced and Remote User Studies for Quality of Experience and Usability Research at QoMEX 2021.
Laboratory studies are an established and essential tool for Quality of Experience (QoE) and User Experience (UX) research. However, they required well-equipped test rooms and personnel for the supervision of the test participants. Therefore, they are often cost- and time-intensive. Further, the number of test candidates is often limited due to the sparse laboratory space and need for the test taskers' physical presence in the test environment. In the past months, the pandemic situation made it even more challenging to conduct laboratory studies by increasing the organizational overhead and limiting potential participants. Two possibilities to overcome the current situation are crowdsourcing and remote user studies.
In this context, the special session aims to foster contributions following the traditional way of optimizing and designing crowdsourced subjective studies for Quality of Experience and User Experience research and additionally wants to raise awareness and foster research in a new research direction, the Quality of Experience and User Experience of crowdsourcing workers. The special session also seeks to encourage researchers to exchange their experiences in remote user studies, with non-anonymous test takers, how best practices from crowdsourcing studies can be applied in this context, and discuss which new challenges arise.

The topics of interest include but are not limited to the following:
• Crowdsourcing for subjective studies
• Novel applications
• Limitations of current crowdsourcing systems
• Quality control mechanism and reliability metrics
• Large scale crowdsourcing studies and diversity of participants
• Reproducibility of results
• Reproducibility and cross platform studies
• Assessment of hidden influence factors / Impact of hidden influence factors
• Bias estimation and bias reduction
• Automation and workflows
• Standardization of crowdsourcing test methods
• Usability and User Experience of crowdsourcing tasks
• Optimization of task interfaces and task workflows
• Relation to result quality and worker motivation
• Enhancing workers’ UX (e.g., by means of gamification of tasks)
• Quality of complex crowdsourcing workflows (e.g. combination of AI and Crowds)
• Interconnection of crowdsourcing and lab-based tests,
• Studies comparing results from lab and crowdsourcing
• Adaptations of established lab-test standards to the crowdsourcing environment
• Remote user studies
• Supervised remote user studies
• Remote studies with non-anonymous users
• (Crowdsourcing) best practices for remote user studies

Authors are kindly invited to submit their full papers by the deadline. All submissions are handled through the QoMEX2021 submission process:
For any query, Please write mail to babak.naderi@tu-berlin and

Sincerely Yours,

Matthias Hirth, Institute of Media Technology, TU Ilmenau,
Babak Naderi, Quality and Usability Lab, Technical University of Berlin

Grant details:

In line to support research on the QoE using crowdsourcing, offers $200 grants for selected ideas who want to conduct an experiment on their platform and submit a paper to this special session. The applicants have to send a research abstract and a rough cost plan how the grand will be spend to the to the session organizers. All applications will be forwarded to where a company representative will make the final funding decision. Note, the funds cannot be withdrawn from Microworkers and can only be used for research purposes.
Please send your 1)research abstract, 2) cost plan by20th January (AOE) to and The selected candidates will be announced on 21th of January.

Related Resources

ETSI STQ workshop 2022   ETSI STQ Workshop - Quality of Emerging Services for Speech and Audio: A user-centred perspective
IUI 2023   Intelligent User Interfaces
UMAP 2023   User Modeling, Adaptation, and Personalization
GWS 2023   10th International Conference on Gender & Women's Studies 2023
IJHAS 2022   International Journal of Humanities, Art and Social Studies
CGWS 2023   Canadian International Conference on Gender & Women's Studies 2023
ICRSG 2023   2023 International Conference on Remote Sensing and Geographic Information (ICRSG 2023)
IEEE HMData 2022   IEEE HMData 2022: Sixth IEEE Workshop on Human-in-the-Loop Methods and Future of Work in BigData
UMAP 2023 LBRD 2023   UMAP ’23: 31st ACM Conference on User Modeling, Adaptation and Personalization: Call for Late-Breaking Results and Demos
PCTS13- 2023   13th International Conference on Professional Communication and Translation Studies