We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byGerman Furniss
Modified about 1 year ago
- 1 -© FTW 2014 WG2 TF Crowdsourcing CROWDSOURCING 2.X From Microworkers to Customers: Lessons learned from crowdsourcing testing Bruno GARDLO, FTW 8 th General Qualinet Meeting, 7.10. – 10.10.2014, Delft
- 2 -© FTW 2014 Simple web-app, with flash player and a ACR 5 rating scale, 2 CCs ranging from 800 to 2000 kbps Volunteers gathered via Facebook and Email calls, 114 users, 3 months of collecting the data Each respondent was directly approached via chat, when he started and finished the test (pseudo-controlled environment) Users rated 10 videos in a row, repeated twice – overall assessment time ~12 minutes Beginnings…Back in the year 2010
- 3 -© FTW 2014 Facebook Study 2010
- 4 -© FTW 2014 Facebook study 2011 Procedure was taking rather long time, so we looked for alternatives, but still focused on Facebook volunteers Enhanced web-app, to be more “standalone” – added content questions, shortened testing time to be more “appealing” to the subjects Reduced administrative burden (no chatting with respondents…) 220 subjects ~ only 812 reliable answers
- 5 -© FTW 2014 Facebook Study 2011
- 6 -© FTW 2014 2012: Move to Microworkers.com o Introduced 2-stage design, together with “Screen quality test” o Additionally – application monitoring, Control questions concerning the playback, Consistency questions, “Gold” data: o Initial number of users: 297 o Number of users remained after screening: 88 (29%)
- 7 -© FTW 2014 2012: Move to Microworkers.com
- 8 -© FTW 2014 2012: Move to Microworkers.com o Introduced 2-stage design, together with “Screen quality test” o Additionally – application monitoring, Control questions concerning the playback, Consistency questions, “Gold” data: o Initial number of users: 297 o Number of users remained after screening: 88 (29%) o Next steps: Improving the efficiency and reliability of the campaigns
- 9 -© FTW 2014 2013: In-Momento Approach Engage user’s attention with interesting easy to do task (and monitor reliability) Retain user’s attention with short testing session and with simple UI Investigate user’s reliability and find a communication way to create a dialog with him – Lead him in the app and inform about progress and how is he doing… Once user is in the app and is reliable -Try to engage him with more tasks -Do not force him to continue -Offer him some advantage over other users (higher payment)
- 10 -© FTW 2014 2013: In-momento Results 100 reliable ratings in several hours (instead of days) Overall reliability: 86% Increased cost efficiency: 0.26 $ vs. 0.08$ per reliable task Decrease of administrative overhead! MOS still higher than that achieved in the Lab
- 11 -© FTW 2014 Facing the CS issues There is a consistency in the results for all our past CS studies – MOS in CS assessment is higher than those achieved in the Lab Some tasks related to the QoE, are easier to recognize for the workers -e.g. users are more familiar with the stalling video, than with recognizable artifacts related to the encoding -Users are accustomed to certain quality level they receive The relation between Lab Studies and CS study results is often not easy discoverable, maybe even not existent
- 12 -© FTW 2014 THANK YOU FOR YOUR ATTENTION
WG2 Task Force “Crowdsourcing” Tobias Hossfeld, Matthias Hirth, Bruno Gardlo, Michal Ries, Sebastian Egger, Raimund Schatz, Katrien de Moor, Christian.
WG2 Task Force “Crowdsourcing” Tobias Hossfeld, Patrick le Callet WG2 Mechanisms and Models https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd:patrasmeeting2013.
AGES: Activating and guiding the engagement of seniors Intervention Training.
“Over Easy” Flipped Experience: Preparing Faculty To Effectively Implement Flipped Teaching Ryan Banow - The Gwenna Moss Centre.
Tshilidzi Tshiredo. Introduction Long time ago even before technologies, social networking platforms and mobile devices, Dewey, J.( ) stated that.
Direct Instruction Meera Mehtaji. Direct Instruction The teacher explains the main concept. Then the teacher guides the students through a problem. The.
Online, Remote Usability Testing Use web to carry out usability evaluations Two main approaches agent-based evaluation (e.g., WebCritera) model automatically.
Measured Progress © New Mexico SBA iTester 3 Webinar Administration Best Practices This presentation will begin momentarily... Please call in by.
What is a VLE? The Ferl Practitioners’ Programme Transforming Teaching and Learning with ILT U4.3a.
Communication for Change The growing demand from our customers Andrea Henry Director, Strategic Communications.
Market research is the process of gathering information which will make you more aware of how the people you hope to sell to will react to your current.
Using CIITS to Create Classroom Assessments Copyright © 2011 Schoolnet, Inc. All rights reserved.
MICROSOFT RD WEB ACCESS 2008 R2 Remote Desktop Services Copyright 2012 Interfaceplanet.com | Phone: | Design.
BSBIMN501A QUEENSLAND INTERNATIONAL BUSINESS ACADEMY.
Indicators of Public Administration Capacity Building for ESF monitoring.
Confidential ©Nokia 2008 Innovation and Monetization of Mobile Applications Sunil Rao Head of Forum Nokia -India.
Results from the User Survey Tobias Hossfeld WG2 TF„Crowdsourcing“ https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowdhttps://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd.
Going Hybrid – part 2 Moving to Hybrid Cloud with Windows Azure Virtual Machines & System Center 2012 R2.
Marketing CH. 4 Notes. The Basics of Marketing Marketing today is different from marketing only a few years ago It has expanded from a few activities.
Transitioning Content, Students and Instructors. Illustrate advantages of blended learning Identify major business impacts Examine alternative media available.
The Analysis Phase Objective: To study the current system, organization, and problems to determine what changes need to be made in order to achieve the.
Performance Assessment. Performance assessment is a dynamic process calling for students to be active participants, who are learning even while they are.
TEACHERS AS MODELS FOR DIGITAL LEARNING By Jessica Bonatsos.
How we get more leads for our clients the Snap guide to BS-Free marketing.
Pre-QA activities: Quality planning. (a) Set specific quality goals. (b) Form an overall QA strategy, which includes two sub-activities: Select appropriate.
Chapter 11 Adding Sound and Video. Chapter 11 Lessons 1.Work with sound 2.Specify synchronization options 3.Modify sounds 4.Use ActionScript with sound.
Sagecrm.com/7.2 Introducing Sage CRM 7.2 Mark Green Business Consultant Pinnacle.
Developing Business Data Collection and measuring Response Burden European Conference on Quality in Official Statistics Helsinki 4-6 May 2010 Johanna Leivo,
Flipping for Instruction: Using the Flipped Class Model in Library Instruction Eduardo Rivera Head of Reference Services – LIU Post SUNYLA 2015 June 5,
Certify skills through Microsoft ® Office Specialist Microsoft Office Specialist 2010 represents an exciting opportunity for employees to increase.
What is Usability? Usability Is a measure of how easy it is to use something: –How easy will the use of the software be for a typical user to understand,
Club Matters Introduction to Club Matters. What is Club Matters?
Online Student Success: Teaching the ABCs of Online Proficiency to Produce As, Bs, and Cs in Online Classes.
SSE system Management and Evaluation System, based on the obtained results.
OnlineInset.net Ltd is a non-profit training company limited by guarantee that develops training courses to teach people who support children.
Usability Process for eBP at Intel Eric Townsend, Intel.
MN SW-PBIS Training Kevin Filter School-wide Evaluation Tool (SET) 1.
Institute of Computer Science Chair of Communication Networks Prof. Dr.-Ing. P. Tran-Gia Modeling YouTube QoE based on Crowdsourcing and Laboratory User.
Tutoring Chapter 13. Reflections on Tutoring If your school district is experiencing budget cuts, and the school board wants to end the tutoring program.
Accelerated Reader Questions & Answers Sunrise R IX.
Skills and Techniques Intermediate 2 & Higher Physical Education.
Reducing administrative burden in Bulgaria: Single Entry Point for Reporting Fiscal and Statistical Information Dr.Mariana Kotzeva President of National.
Review an existing website Usability in Design. to begin with.. Meeting Organization’s objectives and your Usability goals Meeting User’s Needs Complying.
Starter Logon Create a folder in your Year 9 ICT folder called “Y9 Flash Animation” Monitors Off Planners out Watch this videovideo.
Certificate in Digital Applications – Level 02 Multimedia Showcase – DA202.
April NOTE: Sign in with Office 365 ID.
Christa Marsh Southern Arkansas University Biology Professor.
Damian Gordon. Summary and Relevance of topic paper Definition of Usability Testing ◦ Formal vs. Informal methods of testing Testing Basics ◦ Five.
Web Design, 5 th Edition 6 Multimedia and Interactivity Elements.
© 2017 SlidePlayer.com Inc. All rights reserved.