1. WG2 Task Force “Crowdsourcing” 9th Qualinet General Meeting, QoMEX 2015 Costa Navarino, 26 th May 2015 Tobias Hossfeld WG2 Mechanisms and Models https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd:qomex2015meeting

2. Research Topics: Crowdsourcing Methodology –Reliability –Motivation & Incentives: scales, impact –Eyetracking –Lab vs crowd –Rating scale design –Best practices Application –HTTP adaptive streaming –Multimedia synchronization –Ultra-high-definition DASH –Image aesthetic appeal –Web QoE –Speech quality 2 WG2 TF„Crowdsourcing“ https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowdhttps://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd

3. Highlights (1)Qualinet Recommendation Paper on Crowdsourcing: "Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force “Crowdsourcing”. (2)CrowdMM 2015 Workshop @ ACM Multimedia 2015: fourth time. Chairs: Judith Redi (TU Delft), Guo-Jun Qi (University of Central Florida), Stevan Rudinac (University of Amsterdam) (3)Special Issue on “Crowdsourcing” in Computer Networks Journal (Elsevier) 2015. Guest editors: Tobias Hoßfeld (University of Duisburg-Essen), Phuoc Tran-Gia (University of Würzburg), Dr. Maja Vukovic (IBM Thomas J. Watson Research Center Yorktown Heights) (4)Special session on "Advanced Crowdsourcing for Speech and Beyond" at Interspeech 2015, Dresden, Germany, Sep 2015. Main organizer from Qualinet: Tim Polzehl (TU Berlin) 3 WG2 TF„Crowdsourcing“ https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowdhttps://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd

4. Recent Publications and Results: Oct 2014 - May 2015 1.Babak Naderi, Ina Wechsung, and Sebastian Möller. 2015. Effect of Being Observed on the Reliability of Responses in Crowdsourcing Micro-task Platforms. Quality of Multimedia Experience (QoMEX), 2015 Seventh International Workshop on (2015). 2.Hossfeld and Redi. Journey through the crowd: Best Practices and Recommendations for Crowdsourced QoE, QoMEX 2015 3.Tobias Hoßfeld, Matthias Hirth, Judith Redi, Filippo Mazza, Pavel Korshunov, et al. Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force “Crowdsourcing”. Lessons learned from the Qualinet Task Force “Crowdsourcing” COST Action IC1003 Qualinet, 2014. 4.Hoßfeld, Tobias, Michael Seufert, Christian Sieber, Thomas Zinner, and Phuoc Tran-Gia. “Identifying QoE optimal adaptation of HTTP adaptive streaming based on subjective studies.” Computer Networks 81 (2015): 320-332. 5.Martín Varela, Lea Skorin-Kapov, Toni Mäki and Tobias Hoßfeld. QoE in the Web: A Dance of Design and Performance, QoMEX 2015 6.Pierre Lebreton, Evangelos Skodras, Toni Mäki, Isabelle Hupont Torres, and Matthias Hirth. Bridging the gap between eye tracking and crowdsourcing. Proceedings of the SPIE 9394, Human Vision and Electronic Imaging XX. San Francisco, USA, February 2015 7.Judith Redi and Isabel Povoa. Crowdsourcing for Rating Image Aesthetic Appeal: Better a Paid or a Volunteer Crowd? 3rd International ACM workshop on Crowdsourcing for Multimedia (CrowdMM 2014), Orlando, USA, November 2014 4 WG2 TF„Crowdsourcing“ https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowdhttps://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd

5. Recent Publications and Results: Oct 2014 - May 2015 (f.) 8.Babak Naderi, Ina Wechsung, Tim Polzehl and Sebastian Möller. Development and Validation of Extrinsic Motivation Scale for Crowdsourcing Micro-task Platforms. 3rd International ACM workshop on Crowdsourcing for Multimedia (CrowdMM 2014), Orlando, USA, November 2014 9.Christian Timmerer, Touradj Ebrahimi, Fernando Pereira, Toward a New Assessment of Quality, In IEEE Computer, IEEE Computer Society, vol. 48, no. 3, Los Alamitos, CA, USA, pp. 108-110, 2015. 10.Benjamin Rainer, Stefan Petscharnig, Christian Timmerer, Merge And Forward - Self-organized Inter-Destination Multimedia Synchronization, In Proceedings of the 6th ACM International Conference on Multimedia Systems (available not, ed.), ACM International Conference on Multimedia Systems, New York, U.S.A, pp. 4, 2015. 11.Benjamin Rainer, Stefan Petscharnig, Christian Timmerer, Is One Second Enough? - Evaluating QoE for Inter-Destination Multimedia Synchronization using Human Computation and Crowdsourcing, In Seventh International Workshop on Quality of Multimedia Experience (QoMEX 2015) (Athanassios Skodras, ed.), IEEE, Greece, Messinia, pp. 1-4, 2015. 12.Benjamin Rainer, Christian Timmerer, Quality of Experience of Web-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing, In First International Workshop on VideoNext: Design, Quality and Deployment of Adaptive Video Streaming (N N, ed.), ACM, Australia, Sydney, pp. 1-6, 2014. 13.Benjamin Rainer, Christian Timmerer, Self-Organized Inter-Destination Multimedia Synchronization For Adaptive Media Streaming, In Proceedings of the 22st ACM International Conference on Multimedia (ACM, ed.), ACM, New York, NY, USA, pp. 10, 2014. 5 WG2 TF„Crowdsourcing“ https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowdhttps://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd

6. Current Event related Activities (1)Workshop on "Visual Crowds for Geo- and Environmental Sciences", Heidelberg, Germany, Dec 2014: Strong representation by Qualinet Crowdsourcing TF members (Judith Redi, Pavel Korshunov, Matthias Hirth, Tobias Hossfeld) (2)Dagstuhl Seminar “Evaluation in the Crowd: Crowdsourcing and Human-Centred Experiments”, Nov. 2015: Daniel Archambault (Swansea University, GB), Tobias Hoßfeld (Universität Duisburg- Essen, DE), Helen C. Purchase (University of Glasgow, GB), Maja Vukovic (IBM TJ Watson Research Center – Yorktown Heights, US) (3)Tutorial “Adaptive Media Streaming and Quality of Experience Evaluations using Crowdsourcing”: Christian Timmerer, Tobias Hoßfeld (4)Tutorial “Over-the-Top Content Delivery: State of the Art and Challenges Ahead”: ICME2015, Christian Timmerer, Ali C. Begen 6 WG2 TF„Crowdsourcing“ https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowdhttps://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd

7. Current Activities: Research Topics Robustness in Speech Quality Assessment and Temporal Training Expiry in Mobile Crowdsourcing Environments: Tim Polzehl, Babak Naderi, Friedemann Köster, Sebastian Möller Effect of Trapping Questions on the Reliability of Speech Quality Judgments in a Crowdsourcing Paradigm: Babak Naderi, Tim Polzehl, Ina Wechsung, Friedemann Köster, and Sebastian Möller. Crowdsourcing Work Motivation Scale: Development and validation for crowdsourcing micro-task platforms: Babak Naderi, Ina Wechsung, Tim Polzehl and Sebastian Möller. Object identification with crowdsourcing: Lucjan Janowski, Tobias Hossfeld, Matthias Hirth Crowd vs. lab: lessons learnt from subjective studies on image aesthetic appeal (JOC experiments): Judith Redi, Ernestasia Siahaan, Pavel Korshunov, Julian Habigt, Tobias Hossfeld Rating scale design and training in crowdsourcing: Bruno Gardlo, Sebastian Egger, Tobias Hossfeld Adaptive crowdsourcing user tests with smartphones: Michael Seufert, Tobias Hossfeld Crowdsourcing for user-centric multimedia assessment: Tobias Hossfeld, Touradj Ebrahimi Forum paper (in form of interview of experts) to be initiated by Patrick le Callet Crowdsourced Eye-Tracking Experiments: Pierre Lebreton, Evangelos Skodras, Toni Mäki, Isabelle Hupont Torres, and Matthias Hirth Advanced Ultra-High-Definition DASH: Christian Timmerer with bitmovin 7 WG2 TF„Crowdsourcing“ https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowdhttps://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd

8. WG2 Task Force “Crowdsourcing” 9th Qualinet General Meeting, QoMEX 2015 Costa Navarino, 26 th May 2015 Tobias Hossfeld WG2 Mechanisms and Models https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd:qomex2015meeting