1. U.S. Army Research, Development and Engineering Command * Unclassified – Approved for Public Release* What to Consider When Developing and Evaluating Games for Training Teresita Marie Sotomayor, PhD Science and Technology Manager US Army Research Laboratory Simulation and Training Technology Center

2. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Outline This presentation will explore: – Generational differences and the characteristics of the new generation of trainees –Features of games that motivate and engage individuals – An overview of the elements that need to be considered when evaluating the effectiveness of games in a training environment

3. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Generational Differences When developing games for training it is important to consider nested generational groups within the learning population. Generation X Prefer fast-paced presentation of information Frequent feedback Sense of accomplishment Visual – grown up in the 3Dworld Single channelers Generation Z Born with complete technology Highly creative and collaborative Multi-taskers React fast Generation Y grown up with the internet at their fingertips World to them is virtual Possibilities are endless Multi-channelers Training Environment

4. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Why? When designing tools for training it is important to understand targeted audience: –Characteristics – Learning Preferences – Cognitive Styles

5. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Sample Population Profile Educational background Active Duty Combat Medics and Reservists: some college Prior level of medical training Active Duty and Reservist: some prior and combat medical training Computer literacy level Active Duty: intermediate level computer literacy Reservists: basic computer literacy Male/female ratio 5 to 1 male to female Age range 18 to 50 level 12 th grade Learning style Visual and Kinesthetic Background Various backgrounds Some foreign students Motivation for completing the training Job requirement, advancement, job knowledge, personal pride

6. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Sample Population Profile Age Range: 18 to 42 Generation X: 13% Generation Y: 87% Gender Male: 73% Female: 27% Male to Female Ratio: 3 to1 Experience Combat Operations: 8% Combat Medic: 2% Civilian Setting: 29%

7. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Sample Population Profile Video Game Experience: 93 % Ability: Beginner: 29% Intermediate: 48% Expert: 23% Computer Ability: Beginner: 11% Intermediate: 72% Expert: 16% Education HS: 30 % Some College: 53% Associate: 10% Bachelor: 6% Post Bachelor: 1%

8. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Gender Generation X Total # (Percentage) Generation Y Total # (Percentage) Male17 (12.9)115 (87.1) Female6 (12.5)42 (87.5) Total23 (12.7)157 (87.3) 93% Sample Population Profile Gender and Generation Categorization

9. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* 93% Sample Population Profile Id #AgeGenerationGenderEducation 1026YFemaleHS 1830YFemaleBachelors 141XMaleBachelors 2622YMaleSome College 840XMaleAssociate 1918YFemaleHS 4719YMaleHS 11520YFemaleSome College 14819YFemaleHS 13827YFemaleBachelors 7523YFemaleSome College 6826YFemaleAssociate 12635XMaleSome College Descriptive Characteristics of Subjects without Video Game Experience

10. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Sample Population Profile Gender First Person Video Game Ability Beginner Intermediate Expert Computer Ability Beginner Intermediate Expert Female32(67%) 15(31%) 1(2%)6(13%) 38(81%) 3(6%) Male21(16%) 71(54%) 40(30%)14(11%) 91(69%) 26(20%) First Person Video Game and Computer Ability Self Assessment

11. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Sample Population Profile Distribution of Video Game Hours on a Typical Week

12. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Games are a form of fun Games are a form of play Games have rules Games have goals Games have outcome and feedback Why Games Engage?

13. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Games are adaptive Games have win states Games have conflict/competition/challenge/opposition Games have problem solving Games are interactive Games are immersive Why Games Engage?

14. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Games are Interactive Interactivity in Games reinforces: -Individual Control -Trial-and-Error -Constant Change Why Games Engage?

15. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Instructional Games Instructional Games: –Designed or modified to meet training objectives –Includes rules, constraints, and activities that closely replicate the constraints of the real world tasks being trained –Must be incorporated in the POI in a manner that ensures that trainees understand the training objectives and receive feedback on performance against the training objectives

16. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Instructional Games Instructional Games: –Should be considered as a training aid tool –If developed as a stand-alone training tool, the game must be designed such that all instructional capabilities provided by the trainer are incorporated –Feedback should be incorporated as a structured guided activity such that learning can be fostered and the experience can be meaningful to training

17. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Instructional Games Four main aspects of a “serious game”: 1.Context of the game and usage is essential to the effectiveness of the game in the training environment 2. Learner Specification – user profile to understand characteristics, learning preferences and cognitive styles of the targeted group of learners

18. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Instructional Games Four main aspects of a “serious game”: 3. Representation – how levels of immersion, fidelity, and interactivity are integrated such that it is effective and provides the required level of engagement 4. Pedagogical model – learning theory and approaches behind the application to ensure that it is used effectively to support specified learning

19. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* When Designing a “Serious Game” USER Centric Approach - Gap Analysis and Requirement Definition Structured SME Discussions Observation of training to ID objectives and gaps in training Training Curriculum Review Task Analysis - Refinement Usability Studies - Assessment Training Effectiveness Evaluation

20. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* The usability of an interactive system can be defined, in part, in terms of how easily a user can access and use the intended functionality of the system to meet task objectives. Usability Studies

21. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Interactive systems should be designed such that they are intuitive, effective, and subjectively acceptable to users. Performing a usability study early on can provide valuable information for system development. Usability Studies

22. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Early identification of user issues translates into reduced system redesign costs, enhanced user satisfaction and decreased user training time and the need for technical support. Usability Studies

23. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* User Centric Methodology utilized to: - Gather user (instructors and potential users) feedback regarding the current state of the technology being developed - Explore the feasibility of incorporating innovative training tools in the current program of instruction Usability Studies

24. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Observation of trainees and trainers interacting with the system Feedback on their reactions captured through surveys and participation in structured focus group interviews. Methodology

25. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Questionnaires are effective instruments to collect: – Demographic data – Feedback regarding their experience with the Simulation – Reaction to training Methodology

26. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Usability questionnaires are effective instruments to gather subjective data regarding user’s reaction: - Benefit to Training, System Usability, Presence and Realism, and Motivation to Use -Training Objectives, Knowledge and Skills, Overall Reaction to Training Methodology

27. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Focus groups are effective instruments to gather collective subjective data that can’t be collected via survey -Feedback about the current state of the prototype - Scenario/Environment, Techniques, User Interface, Training Objectives and After Action Review (AAR) Methodology

28. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Training Effectiveness Evaluation The main purpose of a training program is to impart the acquisition of knowledge, skills, and competencies. Evaluating a particular training is a challenging task

29. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Training Effectiveness Evaluation Developed by Donald Kirkpatrick in 1952 and published in 1959 in the Journal of the American Society of Training Directors Provides a methodology for evaluating training programs It is widely used because is simple and practical and can be applied to different situations (is flexible)

30. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Level I Reaction Criteria Reaction: Evaluates user affective and attitudinal response to the training program. Focuses on self-report measures. Instruments: Survey, questionnaires, focus groups, etc. Level II Learning Criteria Knowledge Acquisition: Evaluates user’s increase in knowledge and capability as a result of the training program. Focuses on learning outcomes. Instruments: pre-/post Knowledge Test, Interview or on-the-job assessments, etc. Level III Behavior Criteria Skill Transfer: Evaluates changes in behavior on the job as a result of the training program. Focuses on the job performance measures. Instruments: self-assessments, on-the-job performance ratings, etc. Level IV Results Criteria Benefits: Evaluates the effect on the organization resulting from the training program. Focuses on long-term productivity performance measures. Instruments: Utility Analysis Instruments, Organizational performance reports, QA Reports, etc. Kirkpatrick's Model

31. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Level 1 Reaction Measure of how participants feel about the various aspects of a training program. Why should we measure reaction? -People must like a training program to obtain the most benefit. -To ensure participants are motivated and interested in learning.

32. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Level 1 Reaction Guidelines: - Determine what needs to be found out about the training - Develop Instruments: -Questionnaires, focus groups, researcher observations on participants reaction to the simulation - Need to ensure that reactions can be tabulated and quantified - Obtain honest reactions by making the sheets anonymous

33. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Level 2 Learning This is a measure of the knowledge acquired, skills improved, or attitude changed due to training. In our context we are aiming to improve knowledge of concepts, principles, or techniques. Why? -Measure of reaction of trainees is not an indication that learning has taken place. - Will indicate what principles, facts, and techniques were understood and absorbed by trainees.

34. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Level 2 Learning Guidelines: - Use a Control Group - Develop instruments to evaluate knowledge before and after training: Pre/post test, assessments, ratings of performance - Use 100 percent scale - Evaluate results statistically so that learning can be proven - Use the results of the evaluation to take appropriate action

35. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Level 3 Behavior This is a measure of the extent to which participants change their on-the-job behavior because of training. It is also referred as transfer of training. Why? - There is a big difference between knowing principles and techniques and properly using them on the job.

36. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Level 3 Behavior Guidelines: - Use a Control Group - Allow enough time for change in behavior to take place - Develop quantitative instruments: Scenario based simulations, self efficacy instruments, on-the-job observations - Survey or interview one or more of the following: trainees, instructors, superiors, etc - Use the results of the evaluation to take appropriate action

37. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Measure of the long term results of the training program in terms of reduced costs, higher quality, higher productivity, lower rates of employees turnover or absenteeism, etc. Why? It can be used to support acquisition of a specific training device or to include methodology in POI. Warning: Evaluations become more difficult, complicated and expensive. How much of the improvements shown over time are due to training and not other variables. Level 4 Results

38. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* Level 4 Results Guidelines: - Use a Control Group - Allow enough time for results to be achieved - May be obtained from unit performance reports, quality inspections and interviews with Senior people - Use the results of the evaluation to take appropriate action

39. SFC Paul Ray Smith Simulation & Training Technology Center Unclassified – Approved for Public Release* ?? QUESTIONS ?? Teresita.sotomayor@us.army.mil