Video Enhanced VR for Teaching Restaurant Skills to Children with Autism

Abstract While Virtual reality has been used for various training applications, few studies have measured its effectiveness in teaching social interactions. This research used web delivered gaming technology to create virtual worlds where children with Autism Spectrum Disorder (ASD) interacted with avatars to practice restaurant social skills. Videos of similar real world situations within the virtual sequences reinforced the lessons and aided in generalization, a known problem for individuals with ASD. Before training, two unknown restaurant social skills were identified for each of five children with ASD, aged 7 to 16. All children who completed VR training correctly performed two new social skills in a virtual restaurant. In post- training real world restaurants, these children exhibited at least one new appropriate social interaction and 75% demonstrated 2 new social skills.

Learning Steps for Autism Define a common social skill deficit-eating in a restaurant Divide social skill into discreet steps (1) Wait to be seated, (2) Answer hostess questions, (3) Follow hostess to table, (4) Wait for server and read menu, (4) Order food, (5) Table conversation Define measurable actions in each step Ex: Step 1 includes determining if there is a hostess or you seat yourself, waiting in the appropriate place, recognizing the hostess, and waiting for others already in line Identify meaning behind each step - Theory of Mind Show a range of appropriate responses for each step Use non-threatening practice of steps with avatars in VR Display video of real people in same step within VR lesson Reward or correct at each step based on childs actions Vary scene/avatar actions to teach variations Allow user control of steps for different learning abilities Vary avatar responses to avoid patterns and teach concepts

Program Design Text with video explanation of each step What to do (Question/How to Respond) Explain the social action or exchange Why? -Explain the meaning behind the actions See it! -See a video of real people doing the step Oops… What to do if something goes wrong Show possible variations of What to do Virtual world practice in pizza restaurant Three windows on screen: Interactive VR world with avatars Question/answer and video/correction window Displays responses child can pick If child picks correct response, shows video If child picks wrong response, corrects/repeats step What to do/Why? Guidance Multi-player practice with other people over web

Program Details Player moves in virtual worlds with mouse or keyboard directional keys Avatar program tracks and verbally responds to different actions of the player Avatars in virtual world include player, a friend the player is eating with, a hostess, a waitress, and other patrons at different tables Visual cues. Example: Colors visually separate window functions; Error/reward statements in red, questions with choices in blue, inactive window in black Videos at each step are repeatable with button control Avatar spoken words have both text displayed at window bottom and audio Print out of step information available for guidance if child prefers hard copy Text with video explanation program was developed with Flash VR practice world was created and delivered over web with Wild Tangent Platform Multi-player VR practice was created and delivered over web with Adobe Atmosphere VR actions are controlled with JAVA programs Graphics used Wild Tangent world creation toolkit, Atmosphere world creation toolkit, 3d StudioMax, Poser, and Photoshop Video production was done in a real a restaurant during lunch hour. Waitress, hostess, and friend were scripted actors. The regular restaurant clientele was part of the filming background. Fourteen action and seventeen avatar phrase variations were used Video processing was done with Premier

Study Design Six children ages 7 to 16 with ASD chosen All in special education programs and recommended by therapists. Child 01(Age 11, Male, Asian)-Child 02(Age 7, Male, White)-Child 03(Age 9, Female, White)-Child 04(Age 16, Male, Black)-Child 05(Age 9, Male, White)-Child 06(12, Female, Black). Pre-training testing in real restaurants on six skill steps Individual testing was done in multiple lunch or dinner visits. Restaurants varied and depended on parent recommendation. Interactions were not scripted. Actions and latency were recorded by 2 observers. Correct response required that 90% of all parts of a step be done correctly. Child had to display inappropriate behavior at more than chance level for 2 or more steps to be included in study. Five children practiced correct social actions on computer Training was approximately one hour, two to three times over a 2 week period. Training first involved a treasure hunt game to practice navigation in virtual space. All children mastered control in no more than two plays. Each child then completed the Text, VR Pizza world, and web training programs. After training, each child assessed his knowledge with a written questionnaire. All children came back approximately one week later and repeated the three training programs and questionnaire. Post-training testing in real restaurants on six skills Pre-training testing was repeated in the same restaurants and data collected as before. Appropriate skill responses varied between visits because hostess, waitress, patrons, and other details varied, creating different required social interactions. Two restaurant visits were done per child with an average of 3-day separation between visits.

Conclusions 4 children completed study Child 03 knew all skills in pre-tests. Child 06 was removed because of family conflicts All learned 2 new skills in virtual worlds at 100% accuracy Four subjects mastered all 6 skill steps to 100% accuracy on all computer training All did 1 new skill in real world with 90% accuracy 75% did 2 new skills in real world with 90% accuracy Skill retention was strongest when the restaurant visit was close in time frame to computer practice. Child 05 did a third computer training between restaurant visits after missing Step 1 in first post-training and subsequently did all skills correctly Mixed results for one child who generalized only 1 skill Child 04 was severely autistic (CARS 34) and missed all skills in pre-training. In first post-training, he demonstrated 100% accuracy in 5 of 6 steps. In second restaurant visit, music and the unexpected presence of teacher at next table distracted child. Computer refresher and retest could not be completed before child moved to another state Problems Real world actions did not always match discreet step divisions. Example: child might order correctly but stare at ceiling rather than waitress (was given 50% accuracy). Observers required that each action be completely acceptable by normal social standards Graphs of results and details of pilot study available from