1. CRESST ONR/NETC Meetings, 17-18 July 2003 17 July, 2003 ONR Advanced Distributed Learning Greg ChungAllen Munro Bill BewleyQuentin Pizzini Girlie DelacruzUSC/BTL Terry Vendlinski Zenaida Aguirre-Munoz UCLA/CRESST USN EDO Application  2003 Regents of the University of California

2. CRESST ONR/NETC Meetings, 17-18 July 2003 1 CRESST/BTL/USN EDO Collaboration Research to develop new kinds of assessment and instructional tools to support USN EDO school Initial domain: Risk management within the integrated case study Tool prototypes: –Knowledge mapping (assessment) –Decision-analysis (instruction)

3. CRESST ONR/NETC Meetings, 17-18 July 2003 2 Assessment Focus Knowledge mapping as a way to measure problem solving –Examine students’ understanding of the decision analysis process –Examine the extent to which domain-independent links can be used to measure problem solving processes

4. CRESST ONR/NETC Meetings, 17-18 July 2003 3 Context Integrated Case Study (ICS) –Problem-based learning –Teams are given broad requirements –Deliverable is a professional- level briefing Problem solving, teamwork, communication, prior job experience –Concepts covered in instruction are put into practice in ICS

5. CRESST ONR/NETC Meetings, 17-18 July 2003 4 Domain Risk Management –Complex — multiple variables to consider, broad requirements, imperfect information, changing conditions –No “correct” answer — trade-offs among cost, schedule, and performance variables –Reasoning and analytical processes are critical to success

6. CRESST ONR/NETC Meetings, 17-18 July 2003 5 The Tools Knowledge maps –Students depict their understanding of the decision process underlying the refueling at sea (RAS) “hiccup” –Students’ maps can be compared to an expert’s map for assessment purposes Decision-analysis –Tool permits quantitative or qualitative views of problem solution alternatives –Instruction can be delivered in the context of the tool

7. CRESST ONR/NETC Meetings, 17-18 July 2003 6 Decision Analysis Template Options Analysis Selection Domain Variables Environment

8. CRESST ONR/NETC Meetings, 17-18 July 2003 7 Risk Management Environment Selection Options Analysis Cost Schedule Performance

9. CRESST ONR/NETC Meetings, 17-18 July 2003 8 Risk Management: Refueling at Sea Hiccup Existing system Recompete Full inventory buy Technical data plan buy Full + TDP buy Selection Analysis Cost Schedule Performance No. ships Time constraints Config. Mgt Manufacturing Contracting Supportability T&E Options Environment Requirements Regulations Legal issues

10. CRESST ONR/NETC Meetings, 17-18 July 2003 9 Assessment Tools How do we measure an EDO’s problem solving processes within the risk management domain? –Knowledge maps –Online processes and decisions with decision-analysis simulation tool

11. CRESST ONR/NETC Meetings, 17-18 July 2003 10 Refueling At Sea (RAS) Scenario

12. CRESST ONR/NETC Meetings, 17-18 July 2003 11 Refueling At Sea (RAS) Scenario Options Analyze Select Schedule Cost Performance Environment

13. CRESST ONR/NETC Meetings, 17-18 July 2003 12 Refueling At Sea (RAS) Scenario

14. CRESST ONR/NETC Meetings, 17-18 July 2003 13 Current Status Instructor consensus knowledge map created for RAS scenario Domain-independent links –Harder for processes (vs. declarative) –Level of abstraction issues Domain-independent: context Domain-dependent: requirement, legal, regulation

15. CRESST ONR/NETC Meetings, 17-18 July 2003 14 Next Steps Pilot test at EDO with EDO instructor and EDO class (July/Aug) Evaluate and revise for next class

16. CRESST ONR/NETC Meetings, 17-18 July 2003 15 Decision Aiding Tool http://btl.usc.edu/WS/rasProb/rasProb.jnlp

17. CRESST ONR/NETC Meetings, 17-18 July 2003 16 Decision Making and Risk Management A major aspect of EDO education The Refueling at Sea (RAS) problem –Evaluating options systematically A tool that promotes systematic assessments of risks and remediations A tool that teaches about itself and about risk and decision-making

18. CRESST ONR/NETC Meetings, 17-18 July 2003 17 Decision Aiding Tool Demo TDP = Technical Data Package T&E = Test and Evaluation RAS = Refueling at Sea system Decision Thresholds Controller Decision Thresholds Controller User created nodes have arbitrary names. User can set estimated probabilities, utilities. RAS = Refueling at Sea system

19. CRESST ONR/NETC Meetings, 17-18 July 2003 18 Decision Aiding Tool Instruction

20. CRESST ONR/NETC Meetings, 17-18 July 2003 19 Analyzing the User Application of Chung’s ‘Click- through’ interface.

21. CRESST ONR/NETC Meetings, 17-18 July 2003 20 General Application Instructional approach can provide opportunities for students to engage in systems engineering thinking Assessment approach can provide information on what people know and understand about processes