1. Bayesian Hierarchical Models of Individual Differences in Skill Acquisition Dr Jeromy Anglim Deakin University 22 nd May 2015

2. Functional form of the learning curve Researchers have long been interested in functional form of the learning curve – Power law of practice (Newell and Rosenbloom, 1981; Snoddy 1926) – Evidence for exponential function at individual level (Heathcote, Brown, & Mewhort, 2001) Early example: 1024 choice-reaction time task Data from Seibel 1963; shown in Delaney et al 1998 Early example: 1024 choice-reaction time task Data from Seibel 1963; shown in Delaney et al 1998 Task Results

3. Relating subtask to overall task learning Issue of how to integrate basic findings from cognitive psychology with learning on more complex tasks Lee and Anderson (2001) proposed reducibility hypothesis suggesting that learning a complex task could be understood as the culmination of learning many component subtasks They also proposed that subtask learning will be consistent across subtasks and follow the power law of practice

4. Lee & Anderson (2001) Overall Task Performance KA Air-Traffic Controller Task Task Analysis Subtask Performance Source: Lee, F. J., & Anderson, J. R. (2001). Does learning a complex task have to be complex?: A study in learning decomposition. Cognitive Psychology, 42(3), 267-316.

5. Gaps / Issues Gaps Reliance on group-level analysis Need to refine definitions and tests of subtask learning consistency Lack of incorporation of trial level strategy use data Approach Need for task that facilitates measurement of strategy use and subtask performance A Bayesian hierarchical approach offers benefits over piece-wise individual-level analysis.

6. Wynton-Anglim Booking (WAB) Task 1. Information Gathering (I) 2. Filtering (F) 3. Timetabling (T)

7. Bayesian Hierarchical Models Increased interest in application of Bayesian Methods in psychology Benefits of Bayesian Approach – Clear and direct inference – Flexible model specification – Range of sophisticated model comparison tools (e.g., DIC, Posterior predictive checks) – Well-suited to modelling repeated measures psychological data (i.e., observations nested within people)

8. Models of Overall Performance

9. Models of Subtask Performance

10. Aims 1.Assess support for power and exponential functions on overall and subtask performance 2.Assess degree of consistency in subtask learning 3.Estimate effect of strategy use on subtask performance 4.Assess degree to which strategy use could explain inconsistency

11. Method Participants – 25 adults (68% female) Procedure – Read WAB Task instructions – Complete as many trials as possible in 50 minutes Processing – Extract strategy use, subtask performance and overall task performance – Trial performance was aggregated into average block performance (15 blocks with approximately equal numbers of trials)

12. Data analytic approach Bayesian hierarchical models were estimated using MCMC methods using JAGS with supporting analyses performed in R Model comparison – Graphs overlaying model fits and data – Deviance Information Criterion (DIC) – Posterior predictive checks

13. 1. Overall performance Does a power or exponential model provide a better model of the effect of practice on overall task performance?

14. Overall performance (group-level)

15. Overall task completion time by block (individual-level)

16. Overall performance: Parameter estimates and model comparison (DIC) Interpretation Power has larger deviance but smaller penalty and smaller DIC Differences are small Interpretation Power has larger deviance but smaller penalty and smaller DIC Differences are small DIC = Mean Deviance + Penalty Rules of thumb for DIC difference: 10+: rule out model with larger DIC 5-10: model with smaller DIC is better DIC = Mean Deviance + Penalty Rules of thumb for DIC difference: 10+: rule out model with larger DIC 5-10: model with smaller DIC is better

17. 2. Subtask performance Does a power or exponential model provide a better model of the effect of practice on subtask performance and what is the effect of constraining subtask learning curve parameters?

18. Subtask performance (group-level)

19. Subtask performance (individual-level)

20. Subtask performance: Parameter estimates Subtask Abbreviations: I = Information Gathering F = Filtering T = Timetabling Subtask Abbreviations: I = Information Gathering F = Filtering T = Timetabling Parameters 1: Amount of learning 2: Rate of learning 3: Asymptotic performance Parameters 1: Amount of learning 2: Rate of learning 3: Asymptotic performance

21. Subtask performance: Model comparison (DIC) Power has lower DIC (3862 vs 3885); but larger mean deviance Constraints substantially damage fit Power has lower DIC (3862 vs 3885); but larger mean deviance Constraints substantially damage fit

22. Subtask performance: Model comparison (posterior predictive checks) Interpretation: When data is simulated from a model and statistics are calculated on simulated data, good models generate statistics similar to actual data Bolding reflects discrepancies Interpretation: When data is simulated from a model and statistics are calculated on simulated data, good models generate statistics similar to actual data Bolding reflects discrepancies

23. 3. Strategy Use on Subtask Performance What is the effect of strategy use on subtask performance?

24. Strategy use (group-level)

25. Strategy use on performance: Parameter estimates Note: Parameter estimates (i.e., exp (lambda)) for strategy covariates on subtask performance exp(lambda): expected multiple to task completion time resulting from strategy use exp(lambda) greater than 1: strategy use increases task completion time exp(lambda) less than 1: strategy use decreases task completion time Note: Parameter estimates (i.e., exp (lambda)) for strategy covariates on subtask performance exp(lambda): expected multiple to task completion time resulting from strategy use exp(lambda) greater than 1: strategy use increases task completion time exp(lambda) less than 1: strategy use decreases task completion time

26. 4. Strategy Use and Subtask Learning Consistency To what extent does strategy use explain subtask learning inconsistency?

27. Strategy use explaining subtask inconsistency (group-level)

28. Strategy use explaining subtask inconsistency (individual-level)

29. Subtask performance with strategies: Model comparison (DIC) Strategies improve fit (e.g., 3885 – 3506 = 379) Damage to DIC fit of constraints is less with strategies (e.g., 3794 – 3506 = 288) than without strategies (e.g., 4497 – 3885 = 612) Strategies improve fit (e.g., 3885 – 3506 = 379) Damage to DIC fit of constraints is less with strategies (e.g., 3794 – 3506 = 288) than without strategies (e.g., 4497 – 3885 = 612)

30. Subtask performance: Model Comparison (Posterior predictive checks)

31. Concluding Thoughts

32. Concluding thoughts Differences between power and exponential are fairly subtle Task learning may be decomposed into subtask learning but functional form of subtask learning can vary Strategy use both expresses learning and learning to trade-off time on subtasks is a strategy itself More generally, the study provides a case study of Bayesian hierarchical methods

33. Future Work Further Bayesian skill acquisition research – Formal models of strategy acquisition – Models of discontinuities in the learning curve – Integrating traits (ability and personality) into dynamic models of performance Extending Bayesian Hierarchical methods to a range of domains – personality faking, longitudinal life satisfaction data, diary employee well-being data

34. Notes Code and data – https://github.com/jeromyanglim/anglim-wynton-2014-subtasks https://github.com/jeromyanglim/anglim-wynton-2014-subtasks Publication – Based on work with Sarah Wynton – Anglim, J., & Wynton, S. K. (2015). Hierarchical Bayesian Models of Subtask Learning. Journal of Experimental Psychology. Learning, Memory, and Cognition. Online First. http://dx.doi.org/10.1037/xlm0000103 http://dx.doi.org/10.1037/xlm0000103 My Contact details – jeromy.anglim@deakin.edu.au – http://jeromyanglim.blogspot.com http://jeromyanglim.blogspot.com

35. Thank you Questions?