1. Crowdsourcing Game Development for Collecting Benchmark Data of Facial Expression Recognition Systems Department of Information and Learning Technology National University of Tainan, Taiwan Hsin-Chih Lin, Zi-Jie Li and Wan-Ling Chu

2. Outline Introduction Literature review Crowdsourcing Game Development Experimental Design and Results 01 02 03 04 Conclusions and Future Works 05 Pacific Neighborhood Consortium Annual Conference and Joint Meetings 20132

3. Introduction Developing an automatic expression recognition system – always use benchmarks Most of facial pictures in benchmarks – not be accepted by the public or other teams Manually classifying facial expression pictures – labor-expensive – time-consuming – difficult to standardize Pacific Neighborhood Consortium Annual Conference and Joint Meetings 20133

4. Literature review Crowdsourcing was first proposed by Howe (2006). The concept of crowdsourcing – to rely on manpower to complete the work – difficult to be replaced by computer programs Microtask & National Library of Finland – Mole Bridge – Mole Hunt Pacific Neighborhood Consortium Annual Conference and Joint Meetings 20134

5. Literature review Von Ahn (2006) proposed the concept of “Games with a Purpose” – attract online players through interactive games “Gamification” can make boring becomes interesting (Krause & Smeddinck, 2011). Listen Game(Turnbull et al., 2007) – improve the results of music search Pacific Neighborhood Consortium Annual Conference and Joint Meetings 20135

6. Crowdsourcing Game Development Pacific Neighborhood Consortium Annual Conference and Joint Meetings 20136 Low Validity Database High Validity Database Benchmark Face Detection Crowdsourcing Game Feature Extraction Classification Face pictures Social classification system social = automatic Automatic recognition system social ≠ automatic expression pictures of low validity expression pictures of high validity

7. Crowdsourcing Game Development 3 by 3 grid – seven pictures – expression hint – two options Game-play rules – two minutes – randomly prompt an expression hint – none of the above – skip Pacific Neighborhood Consortium Annual Conference and Joint Meetings 20137

8. Experimental Design and Results This study enables crowds to classify facial expressions in the game during four-week experiments period – 100 participants – 1,416 times Training and testing method of the automatic expression recognition system : – 80/20 – Incremental training Pacific Neighborhood Consortium Annual Conference and Joint Meetings 20138

9. Experimental Design and Results Pacific Neighborhood Consortium Annual Conference and Joint Meetings 20139

10. Experimental Design and Results Pacific Neighborhood Consortium Annual Conference and Joint Meetings 201310

11. Experimental Design and Results Our study can effectively train automatic recognition system that allows the precision rate of system raised to extremely high in four-week testing. The dual system is able to develop an automatic recognition system in this study. Pacific Neighborhood Consortium Annual Conference and Joint Meetings 201311

12. Experimental Design and Results Our benchmark – 84 happiness – 51 sadness – 34 surprise – 30 anger Pacific Neighborhood Consortium Annual Conference and Joint Meetings 201312

13. Conclusions and Future Works An innovative dual system mechanism – an organism – enhanced the extremely high precision rate of an automatic expression recognition system – efficiency and automation to classification that no matter how many facial expression data needs to be classified – resolve image classification or other issues must through human computation Pacific Neighborhood Consortium Annual Conference and Joint Meetings 201313

14. Conclusions and Future Works Crowdsourcing Game – boring become interesting – save more time and cost – get the classification results agree with crowds Future Works – increase facial pictures – increase expressions categories(disgust, fear, nature) Pacific Neighborhood Consortium Annual Conference and Joint Meetings 201314

15. Thank you for your attention. p6590043@hotmail.com Pacific Neighborhood Consortium Annual Conference and Joint Meetings 201315