1. www.asc-inclusion.eu A project to develop software to assist people with autism to recognise, understand and express emotions through facial expressions, voice and body gestures English only, sorry

2. Introduction A project to develop interactive games to assist people with autism to recognise, understand and express emotions through facial expressions, voice and body gestures Primarily for children aged 5 to 10 Project by 7 organisations under a European Commission funded 3-year cooperative research project (FP7 programme) First version will be available in late 2014 (in English) www.asc-inclusion.eu

3. Autism & understanding emotions Children and adults with autism experience significant difficulties recognising others, as well as their own, emotions and mental states, including facial expressions, vocal intonation, gestures and body language, and their integration in context (Baron- Cohen, 1995; Hobson, 1993; Golan, Baron-Cohen & Golan, 2008). www.ascinclusion.eu www.asc-inclusion.eu

4. Autism & understanding emotions Individuals with autism also experience difficulties in expressing their own emotions non-verbally, including the ability to: – Direct appropriate facial expressions to others (Kasari et al, 1993) – Modulate their vocal intonation appropriately when expressing emotion (McCann & Peppe, 2003) – Use appropriate gestures and body language (Attwood, 1998). – Integrate non-verbal communicative cues with speech (De Marchena & Eigsti, 2010). Understanding and expression of emotions is not learned spontaneously by people with autism, and therefore needs to be taught explicitly. Without these skills, the risk of social exclusion for individuals with autism is significantly increased. www.asc-inclusion.eu

5. Systematic strengths of people with autism Individuals with autism are often hyper-attentive to detail and prefer predictable, rule-based environments and systems. Within such environments, individuals with autism show good, and sometimes even superior systemising skills, compared to the general population (Baron-Cohen, 2003). If provided with a system of emotions, it is plausible that systemising skills could be harnessed to help individuals with autism learn to recognise emotions (Golan & Baron- Cohen, 2006). The computer provides individuals with autism with such a predictable, rule-based environment, enabling them to capitalise on their systemising skills. Computerised socio-emotional training in autism has been shown to change behavior, as well as brain functioning (Bölte et al., 2006) www.asc-inclusion.eu

6. Need for a new software program Studies attempting to teach emotion and mental state recognition to people with autism have shown mixed results, reporting limited generalisation from taught curriculum to situations not included in the training program. Training of individuals with autism requires facilitation by trained specialists, which is not always available, and can be quite expensive. In view of the growing prevalence of autism, there is an urgent need to seek new innovative methods for supporting integration of children with autism into society. Information & communication technologies (ICT), may provide an easily available solution. www.asc-inclusion.eu

7. How the software will work The ASC-Inclusion software will: Analyse the childrens facial expressions, vocal intonations and gestures (using standard microphones and webcams); Train children to recognise their own, and others, facial expressions, tone- of-voice and body gestures through interactive games, text communication, animations, video and audio clips; www.ascinclusion.eu www.asc-inclusion.eu

8. How the software will work The ASC-Inclusion software will: Personalise settings according to childrens individual needs; Support professionals, parents and carers with professional information, reports on childrens progress and forums to interact with other professionals and carers. www.ascinclusion.eu www.asc-inclusion.eu

9. How the software will work The program will consist of 3 state-of- the-art subsystems for: Facial expressions Vocal intonation Body gestures The subsystems will be integrated together in an online virtual world platform to provide the user with a comprehensive and motivating learning experience. www.ascinclusion.eu www.asc-inclusion.eu

10. How the software will work Facial expressions The ASC-Inclusion software will develop this existing system to recognise the users facial expressions. www.ascinclusion.eu www.asc-inclusion.eu Example video of facial expression recognition Example video of facial expression recognition

11. How the software will work Vocal intonation The ASC-Inclusion software will develop this existing system to recognise vocal intonation and adapt it for a learning oriented environment. (Note the unfriendly character used in this example video will NOT be used in the ASC-Inclusion software, only the technology behind it that recognises emotions in the users tone of voice). www.ascinclusion.eu www.asc-inclusion.eu Example video of vocal intonation recognition Example video of vocal intonation recognition

12. How the software will work Body gestures The ASC-Inclusion software will develop this existing system to recognise more subtle gestures of the head, shoulders and hands to express emotions. www.ascinclusion.eu Example video of basic body gesture recognition Example video of basic body gesture recognition www.asc-inclusion.eu

13. Involvement of people with autism www.asc-inclusion.eu Involvement in all stages of software development Direct & indirect involvement (people with autism, as well as their teachers, parents, carers, and professionals) Planning – Information & advice from autism organisations User requirements – Research during early phase of software development to define the needs of people with autism Testing – People with autism directly involved in testing the software in UK & Israel Evaluation – Intermediaries (carers, social workers, psychologists & doctors) involved in evaluating the usefulness of the software for people with autism

14. Status and milestones Estimated start pointMilestones November 2011Project kick off end of year 1 (2012)First version of the basic systems end of year 2 (2013)Experiments with system start end of year 2 (2013)First integrated system end of project (2014)Final project results www.ascinclusion.eu www.asc-inclusion.eu

15. Project partners www.ascinclusion.eu www.asc-inclusion.eu

16. More information & updates www.ascinclusion.eu www.asc-inclusion.eu Website: www.asc-inclusion.eu Join our mailing list / contact us: schuller@tum.de

17. Acknowledgements The research leading to this software product has received funding from the European Communitys Seventh Framework Programme (FP7, 2007-2013), under grant agreement n° 289021. www.ascinclusion.eu www.asc-inclusion.eu