1. CRITICAL THINKING AND DEEP LEARNING: USING NSSE WITH LOCAL SURVEY RESULTS Steve Graunke IUPUI Information Management and Institutional Research Indiana University Higher Education and Student Affairs

2. GOALS What is Deep Learning and what does it have to do with Assessment? IUPUI Principles of Undergraduate Learning Indirect Assessment of SLO’s Background NSSE Deep Learning Scales Using NSSE with Indirect Assessment methods Methodology Relationship between Deep Approaches scales and Indirect Assessment of Critical Thinking Results Voluntary System of Accountability NSSE 2.0 Pedagogy How is this useful?

3. BACKGROUND: WHAT IS DEEP LEARNING? Marton and Säljö (1976) Deep approaches promote better learning Examples of deep approaches Levy Chapman

4. BACKGROUND: WHAT DOES THIS HAVE TO DO WITH ASSESSMENT?

5. DIRECT AND INDIRECT ASSESSMENT Direct Assessment Student Products Indirect Assessment How students feel about their learning

6. BACKGROUND: PRINCIPLES OF UNDERGRADUATE LEARNING Values and Ethics Understanding Society and Culture Intellectual Depth, Breadth, and Adaptiveness Integration and Application of Knowledge Critical Thinking Core Communication and Quantitative Skills

7. BACKGROUND: CRITICAL THINKING  “Definition: The ability of students to analyze carefully and logically information and ideas from multiple perspectives.” “Outcomes: This skill is demonstrated by the ability of students to analyze complex issues and make informed decisions; synthesize information in order to arrive at reasoned conclusions; evaluate the logic, validity, and relevance of data; solve challenging problems; and use knowledge and understanding to generate and explore new questions.” IUPUI Campus Bulletin: http://www.iupui.edu/~bulletin/iupui/2012- 2014/undergraduate/principles.shtml

8. METHODOLOGY: IUPUI CONTINUING STUDENT SURVEY Began in early 1990’s Satisfaction data Expanded over the years Indirect Assessment of PULs

9. METHODOLOGY: NSSE DAL SCALES Scales related to deep learning* Higher Order Learning Integrative Learning Reflective Learning Overall Scale * Nelson Laird, T.F., Garver, A.K., Niskodé-Dossett, A. S., & Banks, J.V. (2008)

10. METHODS: PARTICIPANTS NSSE participants from 2009 CSS participants from 2010 or 2011 105 students

11. OTHER INDEPENDENT VARIABLES DAL use Soft discipline Class level Sex

12. RESULTS: NSSE DAL SCALE ALPHAS Overall Deep Approaches to Learning Scale α = 0.655 Higher Order Learning subscale α = 0.829 Integrative Learning subscale α = 0.631 Reflective Learning subscale α = 0.865

13. RESULTS: NSSE MEANS VariableNMean Std Error of Mean 95% CL for Mean CT (Critical Thinking) 1053.2190.0613.0983.340 fem_flg (Female flag) 1050.6000.0480.5050.694 Sen_flg (Senior flag) 1050.3620.0470.2690.455 soft_flg (soft discipline flag) 1050.6290.0470.5350.722 DAL (Full Deep Approaches Scale) 1052.7490.0512.6492.849 HOL (Higher Order Learning) 1052.9760.0692.8393.114 IL (Integrative Learning) 1052.5790.0502.4802.678 RL (Reflective Learning) 1052.6920.0752.5432.841

14. RESULTS: CORRELATIONS WITH OVERALL DAL SCALE fem_flgSen_flgsoft_flgDAL CT (Critical Thinking) 0.0190.1480.0610.402 fem_flg (Female flag) ---0.0730.3380.147 Sen_flg (Senior flag) -- 0.0870.151 soft_flg (soft discipline flag) -- 0.328 DAL (Full Deep Approaches Scale) --

15. RESULTS: MODEL USING FULL DAL SCALE B β Standard Error t Intercept1.870 0.3695.06* Female-0.010-0.0080.115-0.09 Senior0.1200.0920.1310.92 Soft discipline flag-0.104-0.0800.137-0.76 Deep Approaches to Learning 0.5010.4150.1413.56* F = 6.17 Standard Error of Estimate = 0.582 R squared = 0.175

16. RESULTS: CORRELATION WITH INDIVIDUAL DAL SCALES fem_flgSen_flgsoft_flgHOLILRL CT (Critical Thinking) 0.0190.1480.0610.1950.3880.372 fem_flg (Female flag) ---0.0730.3380.0820.1260.136 Sen_flg (Senior flag) -- 0.0870.1510.1700.052 soft_flg (soft discipline flag) -- 0.2030.2920.281 HOL (Higher-order Learning) -- 0.3250.427 IL (Integrative learning) -- 0.447 RL (Reflective Learning) --

17. RESULTS: CORRELATION WITH INDIVIDUAL DAL SCALES fem_flgSen_flgsoft_flgHOLILRL CT (Critical Thinking) 0.0190.1480.0610.1950.3880.372 fem_flg (Female flag) ---0.0730.3380.0820.1260.136 Sen_flg (Senior flag) -- 0.0870.1510.1700.052 soft_flg (soft discipline flag) -- 0.2030.2920.281 HOL (Higher-order Learning) -- 0.3250.427 IL (Integrative learning) -- 0.447 RL (Reflective Learning) --

18. RESULTS: HIGHER ORDER LEARNING AND INTEGRATIVE LEARNING B β Standard Error t Intercept 1.8700.4014.66 * Female -0.007-0.0060.114-0.07 Senior 0.1030.0790.1360.76 Soft discipline flag -0.088-0.0680.128-0.68 Higher Order Learning0.0680.0770.0970.70 Integrative Learning0.4530.3710.1243.67* F = 6.16 Standard Error of Estimate = 0.588 R squared = 0.166

19. RESULTS: REFLECTIVE LEARNING MODEL B β Standard Error t Intercept 2.3730.2399.95 * Female -0.006-0.0050.122-0.05 Senior 0.1730.1330.1251.38 Soft discipline flag -0.072-0.0560.138-0.52 Reflective Learning0.3090.3820.0853.66* F = 6.12 Standard Error of Estimate = 0.588 R squared = 0.158

20. FINDINGS Relationship between overall DAL and indirect assessment of critical thinking Strong relationship between Integrative learning and critical thinking Strong relationship between Reflective learning and critical thinking Not a relationship between Higher order learning and critical thinking

21. LIMITATIONS No proxy for academic performanceSocio-economic statusIndirect Measures

22. WHAT DOES THIS MEAN? Can encouraging deep approaches improve scores on critical thinking assessments? VSANSSE 2.0Using the DAL scalesUsing NSSE with Indirect Assessment