Formative Research on the Heuristic Task Analysis Process Charles M. Reigeluth Ji-Yeon Lee Bruce Peterson Mike Chavez Indiana University

2 Domain vs. Task Economics Psychology Physics Chemistry Domain Task Management Counseling Engineering Medicine

Domain vs. task expertise Procedural vs. heuristic tasks Heuristic tasks: using principles and guidelines to decide what to do when Heuristics are often tacit knowledge Heuristic Task Analysis (HTA): elicits and formalizes an expert’s heuristic knowledge What is Heuristic Task Analysis?

4 Increasingly complex tasks in society Many tasks are heuristic (continuum) Heuristic task knowledge is often the most valuable and hardest to discover Heuristic task analysis methodologies are underdeveloped Why is HTA important?

5 Knowledge elicitation: Extraction of domain-relevant knowledge from human experts using various techniques Knowledge analysis: Interpretation of the data extracted from experts Knowledge representation: Determined by the purpose of application (e.g. expert system or Artificial Intelligence system) Literature on HTA

6 Most studies deal with procedural rather than heuristic tasks Lack of research that provides guidelines on how to analyze and represent heuristic knowledge Lack of research to inform instructional design

7 The handout shows the version of the Heuristic Task Analysis Process investigated in this research. The Initial HTA Process

8 “ What improvements can be made to the HTA process?” Research Question

9 Developmental research: improves a theory and its real-world application Process : –Apply a theory to an instance –Collect data to improve the application –Suggest improvements to the theory Iterations Three cases Formative Research

10 Task: Group Counseling –Focused on personal growth group –Combination task with both procedural and heuristic elements Constraints: –Due to the confidential nature of counseling, direct observation was impossible –Dealing with interpersonal dynamics which is hard to verbalize –Analyst unfamiliar with group counseling Study I: Context

11 Formative research: designed case Data collection: –Interview: unstructured in-person interviews –Videotapes Participant profile: –3 experts experienced with personal growth group counseling –Expertise ranged from 3 to more than 20 yrs Data analysis & interpretation: –Member checks with participants –Consultation with three other researchers Study I: Methods

12 To best elicit task experts’ heuristic process, we need to incorporate various interview and observation techniques. The analyst’s task expertise needs to be considerable. Working with multiple experts requires certain strategies. Providing reference material during the interview was helpful. Study I: Results

13 Study II: Context Task: Tutoring on Writing –Class of problems focused on higher- level writing concerns such as thesis, unity, and coherence Constraints: –Experts had little time to devote to this project

14 Top-down approach to identify and analyze heuristic knowledge Participant profile: –2 experts experienced in tutoring college students –had passed an evaluation of their tutoring skills each term they had been tutoring Data collection & analysis –Interview: In-person interviews (60 min. each) –Iterative interview & analysis process –Index cards used to record tutor’s responses Study II: Methods

15 Study II: Results Using the top-down process was effective Using index cards was helpful Helping the tutor identify guidelines was problematic Helping the tutor access the context of the tutoring situation was helpful Having the tutor list, prioritize, and choose the decisions made during the tutoring session was effective

16 Task: –Selecting Artwork for Product Line Constraints: –Analyst unfamiliar with business content –Experts had little time to devote to this project Research focus: –Speed and effectiveness of HTA process in business setting Study III: Context

17 Formative research: designed case HTA guidelines converted into lay questions 1 expert, 2 interviews, 1 1/2 hours each Content, process analysis after interviews Member checks w/ 1 content and 1 research expert Study III: Method

18 Speed of HTA process: –Expert was satisfied with time frame –50% of interview was spent eliciting heuristics –1 hour was spent on closely associated tasks –3x5 cards helped analyst get back on track Effectiveness of HTA process: –Expert satisfied when heuristics were elicited –Words like feel, know, like were clues to tacit knowledge and mental models Study III: Results

19 Success with the top-down approach (knowledge categories) varied in the studies More guidance was needed for eliciting the heuristics Using multiple experts presented challenges for the analyst Index cards were helpful for all 3 analysts The HTA process seemed to function well in all three fields. Discussion

20 Preparatory guidance was added to help focus the entire process Guidance was added for multiple experts Guidance added on how to elicit heuristics Bottom-up approach was added (Revisions on handout are in italics) Revisions to the HTA Process

Questions or Comments? For the paper, Charles Reigeluth at