1. Student Goals, Engagement and Outcomes: The Path to Success
Mark Troy
Yunhee Bae
Texas A&M University
AIR Forum, May 21, 2013

2. Goals and engagement are dimensions that facilitate student learning and lead to student success
How are these dimensions related?
Are some more effective than others at facilitating learning?
What combinations of goals and engagement lead to what types of learning?

3. IEO Assessment Model
Input
Engagement
Outcomes

5. Instrument Student Experience in the Research University (SERU)
Administered Spring 2012
42263 students in Six universities
Minnesota
Pittsburgh
Rutgers
South Carolina
Texas A&M
Virginia
7603 responses from TAMU

6. Methods
Exploratory factor analysis and confirmatory factor analysis to specify:
three goal factors (epistemic goals, intrinsic goals, and instrumental goals),
four engagement factors (attention and interest, class preparation, consolidated learning activities, and extra interaction with faculty),
three outcomes factors (knowledge/comprehension, research skills, and communication skills).
Multiple Indicators and Multiple Causes (MIMIC) CFA to validate criterion validities through interrelationships between academic engagement constructs and academic outcomes including the three outcome factors and GPA
Structural Equation Modeling (SEM) to identify relationships between factors

7. I: Inputs Knowledge and skill upon entering the university
Student goals
Epistemic goal
Instrumental goal
Intrinsic goal
Background information
Parents’ education
Parents’ household income

8. Measurement Constructs: Students’ College Goal Orientations (n=6,782)
<Epistemic Goal> Reliability (Alpha) = .64
Confirmatory Factor Analysis
Factor Loadings
Be in a position to give something back to my community after finishing my education.
.55
Integrate spirituality into my life.
.41
Develop a personal code of values and ethics.
.71
Discover what kind of person I really want to be.
.65
<Instrumental Goal> Reliability (Alpha) = .55
Prepare for graduate or professional school.
.57
Achieve a high GPA.
Be in a position to make a lot of money after finishing my education
.45
<Intrinsic Goal> Reliability (Alpha) = .62
Develop an in-depth understanding of a specific field of study.
.52
Explore new ideas.
.58
Acquire a well-rounded general education.
Obtain the skills I need to function in the international arena.
.50

9. E: Engagement Academic Engagement Cognitive Strategies
Attention and interest in class
Class responsibilities
Consolidated learning activities
Extra interactions with faculty
Cognitive Strategies
Elaboration
Critical Thinking
Rehearsal

10. Measurement Constructs: Academic Engagement (n=33,880)
<Attention & Interest in Class> Reliability (Alpha) = .85
Confirmatory Factor Analysis
Factor Loadings
Found a course so interesting that you did more work than was required.
.57
Chosen challenging courses, when possible, even though you might lower your GPA by doing so.
.36
Contributed to a class discussion.
.85
Brought up ideas or concepts from different courses during class Discussions.
Asked an insightful question in class.
.86
Interacted with faculty during lecture class sessions.
.75
Made a class presentation.
.50
<Class Responsibilities >
Class Preparation Reliability (Alpha) = .73
Not Gone to class without completing assigned reading.
.62
Not Gone to class unprepared.
.76
On average, how much of your assigned course reading have you completed this academic year?
Not Skipped class.
.66
Not Turned in a course assignment late.
Time Use Reliability (Alpha) = .53
Studying and other academic activities outside of class.
.77
Attending classes, discussion sections or labs.
.49

11. Measurement Constructs: Academic Engagement (Continued)
<Consolidated Learning> Reliability (Alpha) = .73
Confirmatory Factor Analysis
Factor Loadings
Worked on class projects or studied as a group with other classmates outside of class.
.50
Helped a classmate better understand the course material when studying together.
.57
Sought academic help from instructor or tutor when needed.
.62
Extensively revised a paper at least once before submitting it to be graded.
.61
Raised your standard for acceptable effort due to the high standards of a faculty member.
.53
<Extra-Interaction with Faculty> Reliability (Alpha) = .75
Had a class in which the professor knew or learned your name.
.67
Talked with the instructor outside of class about issues and concepts derived from a course.
.77
Communicated with a faculty member by or in person.
.72
Worked with a faculty member on an activity other than coursework (e.g., student organization, campus committee, cultural activity).
.47

12. Confirmatory Factor Analysis
Measurement Constructs: Emphasis on Cognitive Strategies in the Major (n = 32,034)
<Elaboration> Reliability (Alpha) = .84
Confirmatory Factor Analysis
Factor Loadings
Examined how others gathered and interpreted data and assessed the soundness of their conclusions.
.84
Reconsidered your own position on a topic after assessing the arguments of others.
.75
Incorporated ideas or concepts from different courses when completing assignments.
Used facts and examples to support your viewpoint.
.68
<Critical Thinking> Reliability (Alpha) = .83
Judge the value of information, ideas, actions, and conclusions based on the soundness of sources, methods and reasoning.
.86
Break down material into component parts or arguments into assumptions to see the basis for different outcomes and conclusions.
.82
Create or generate new ideas, products, or ways of understanding.
.71
<Rehearsal> Reliability (Alpha) = .73
Explain methods, ideas, or concepts and use them to solve problems.
.94
Recognize or recall specific facts, terms, and concepts.
.62

13. O: Outcomes Knowledge and Comprehension Skills Research Skills
Communication Skills
GPA

14. Confirmatory Factor Analysis
Measurement Constructs: Students’ Perceived Learning Outcomes (n=28,681)
<Knowledge & Comprehension> Reliability (Alpha) = .82, .80
Confirmatory Factor Analysis
Factor Loadings
At Starting
At present
Analytical and critical thinking skills
.77
.76
Ability to read and comprehend academic material
.75
Ability to be clear and effective when writing
.73
.70
Understanding of a specific field of study
.63
.65
Ability to understand international perspectives (economic, political, social, cultural)
.58 54
Quantitative (mathematical and statistical) skills
.40
.32
<Research Skills> Reliability (Alpha) = .81, .78
Internet Skills
.90
.86
Computer Skills
.84
Library Research Skills
.55
Other Research Skills
.48
<Communication Skills> Reliability (Alpha) = .75, .77
Interpersonal (social) skills
.66
.69
Ability to lead
.67
Ability to prepare and make a presentation
.78
Ability to speak clearly and effectively in English
.51
.56

15. Criterion Validation MIMIC (Multiple Indicators Multiple Causes)
Correlations between factor scores and criteria variables
Parents’ education
Parents’ household income
GPA

16. Correlations Between Factors and Criterion Variables
Correlations between the Factors and Criterion Variables
Path Coefficients from Criterion Variables to the Factors
through MIMIC
GPA
Parent's Annual Income
Parent Education
Multiple R
AE Factors (n=33,880)
Attention & Interest
.12**
.04*
.07**
.00
.05**
.13
Class Responsibilities
.24**
.01
-.03*
.03*
.24
Consolidated Learning
.03**
.02*
-.01
.08
Extra Interaction with Faculty
.13**
.04**
.14
SCG Factors (n=6,782)
Epistemic Goal
-.02
-.05**
-.05*
.05
Intrinsic Goal
.02
-.06**
-.07**
.06
Instrumental Goal
.19**
.20**
-.04*
.19
Note. ** : p < .001; * p < .01

17. Correlations Between Factors and Criterion Variables (continued)
Correlations between the Factors and Criterion Variables
Path Coefficients from Criterion Variables to the Factors
through MIMIC
GPA
Parent's Annual Income
Parent Education
Multiple R
SPLO-S Factors (n=28,681)
Knowledge & Comprehension Skills
.09**
.02**
.13**
.07**
.08**
.16
Research Skills
-.02*
.04**
.05**
-.03**
.03**
.05
Communication Skills
.00
.12**
.06**
.01
.12
SPLO-P Factors(n=28,681)
.20**
.15**
.11**
.18**
.24
.03*
.02*
.03
.14**
.14
ECSM
Critical Thinking
-.01
Elaboration
Rehearsal
.08
Note. ** : p < .001; * p < .01

18. Correlations Between Factors and Criterion Variables
Correlations between the Factors and Criterion Variables
Path Coefficients from Criterion Variables to the Factors
through MIMIC
GPA
Parent's Annual Income
Parent Education
Multiple R
Criterion Variables
GPA
Annual Parents’ Household Income
.13**
Parent Education
.16**
.46**
Note. ** : p < .001; * p < .01

19. Summary
Cronbach’s alphas (.53 ≤α ≤.85; mean = .72) were acceptable for all constructs indicating good internal consistency.
CFAs on each measurement model of AE, SCG, SRLO, ECSM yielded relevant factor loadings (.32 ≤ λ ≤ .94; mean = .64) with good or acceptable model fits, supporting the construct validity of AE, SCG, and SPLO .
The relationship of GPA with “Class Responsibilities” ,”Instrumental Goal”, “Knowledge and Comprehension” at the present, and “Elaboration” were the strongest in AE, SCG, SPLO, and ECSM respectively.
Parents’ annual income is most likely to influence “Knowledge and Comprehension” at present, while, Parent Education is most likely to be related with students’ “Knowledge and Comprehension” at starting indicating a greater impact of parent income than parent education on students’ college learning outcomes
Necessary for the investigation of the relationships among AE, SCG, SPLO, ECSM and GPA to understand the paths to the learning outcomes.

20. Paths to Student Success in the SERU
: The Relationships among 5 domains of students’ Self-Reported Constructs and their cumulative GPA through Structural Equation Modeling (SEM)
To investigate the relationships among Academic Engagement, Students’ College Goal, Learning Outcomes, and Cognitive Strategies. --> How do those constructs lead to each learning outcome including student’s perceived learning outcomes (Knowledge & Comprehension, Research Skills, and communication Skills) and GPA ?
To better understand students’ college experiences and to develop productive policies and programs to heighten students’ competencies for their paths to success.

21. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
Pre-Knowledge & Comprehension
Attention & Interest
Epistemic Goal
Class Responsibilities
Pre- Research Skills
Post-
Knowledge & Comprehension
Intrinsic goal
Consolidated Learning
Pre- Communication Skills
Instrumental Goal
Extra-Interaction with Faculty
Emphasis on Cognitive Strategies in the Major
Elaboration
Critical Thinking
Rehearsal

22. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
Pre-Knowledge & Comprehension
Attention & Interest
Epistemic Goal
Class Responsibilities
Pre- Research Skills
Post-
Knowledge & Comprehension
Intrinsic goal
Consolidated Learning
Pre- Communication Skills
Instrumental Goal
Extra-Interaction with Faculty
Emphasis on Cognitive Strategies in the Major
Elaboration
Critical Thinking
Rehearsal

23. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
Pre-Knowledge & Comprehension
Attention & Interest
R2 = .21
.54
Epistemic Goal
R2 = .01
Class Responsibilities
R2 = .08
Pre- Research Skills
Post-
Knowledge & Comprehension
.59
Intrinsic goal
R2 = .41
.58
R2 = .02
Consolidated Learning
R2 = .18
Pre- Communication Skills
Instrumental Goal
Extra-Interaction with Faculty
R2 = .02
R2 = .15
Emphasis on Cognitive Strategies in the Major
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

24. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Pre-Knowledge & Comprehension
.05
Attention & Interest
-.06
R2 = .21
.18
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
.00
-.01(ns)
-.01
R2 = .01
.13
.02
Class Responsibilities
.00(ns)
R2 = .08
.00
.49
Pre- Research Skills
.16
.02
Post-
Knowledge & Comprehension
.59
.00(ns)
.00
.01(ns)
.16
-.04
.05
.00(ns)
.17
.03
.07
Intrinsic goal
.08
.09
R2 = .41
.09
-.07
.58
R2 = .02
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
-.01
.06
Extra-Interaction with Faculty
R2 = .02
-.03
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01(ns)
.01
.07
.01(ns)
.01
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

25. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Pre-Knowledge & Comprehension
.05
Attention & Interest
-.06
R2 = .21
.18
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
.00
-.01(ns)
-.01
R2 = .01
.13
.02
Class Responsibilities
.00(ns)
R2 = .08
.00
.49
Pre- Research Skills
.16
.02
Post-
Knowledge & Comprehension
.59
.00(ns)
.00
.01(ns)
.16
-.04
.05
.00(ns)
.17
.03
.07
Intrinsic goal
.08
.09
R2 = .41
.09
-.07
.58
R2 = .02
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
-.01
.06
Extra-Interaction with Faculty
R2 = .02
-.03
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01(ns)
.01
.07
.01(ns)
.01
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

26. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Pre-Knowledge & Comprehension
.05
Attention & Interest
-.06
R2 = .21
.18
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
.00
-.01(ns)
-.01
R2 = .01
.13
.02
Class Responsibilities
.00(ns)
R2 = .08
.00
.49
Pre- Research Skills
.16
.02
Post-
Knowledge & Comprehension
.59
.00(ns)
.00
.01(ns)
.16
-.04
.05
.00(ns)
.17
.03
.07
Intrinsic goal
.08
.09
R2 = .41
.09
-.07
.58
R2 = .02
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
-.01
.06
Extra-Interaction with Faculty
R2 = .02
-.03
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01(ns)
.01
.07
.01(ns)
.01
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

27. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Pre-Knowledge & Comprehension
.05
Attention & Interest
-.06
R2 = .21
.18
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
.00
-.01(ns)
-.01
R2 = .01
.13
.02
Class Responsibilities
.00(ns)
R2 = .08
.00
.49
Pre- Research Skills
.16
.02
Post-
Knowledge & Comprehension
.59
.00(ns)
.00
.01(ns)
.16
-.04
.05
.00(ns)
.17
.03
.07
Intrinsic goal
.08
.09
R2 = .41
.09
-.07
.58
R2 = .02
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
-.01
.06
Extra-Interaction with Faculty
R2 = .02
-.03
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01(ns)
.01
.07
.01(ns)
.01
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

28. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Pre-Knowledge & Comprehension
.05
Attention & Interest
-.06
R2 = .21
.18
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
.00
-.01(ns)
-.01
R2 = .01
.13
.02
Class Responsibilities
.00(ns)
R2 = .08
.00
.49
Pre- Research Skills
.16
.02
Post-
Knowledge & Comprehension
.59
.00(ns)
.00
.01(ns)
.16
-.04
.05
.00(ns)
.17
.03
.07
Intrinsic goal
.08
.09
R2 = .41
.09
-.07
.58
R2 = .02
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
-.01
.06
Extra-Interaction with Faculty
R2 = .02
-.03
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01(ns)
.01
.07
.01(ns)
.01
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

29. Post- Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Pre-Knowledge & Comprehension
.05
Attention & Interest
-.06
R2 = .21
.18
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
.00
-.01(ns)
-.01
R2 = .01
.13
.02
Class Responsibilities
.00(ns)
R2 = .08
.00
.49
Pre- Research Skills
.16
.02
Post-
Knowledge & Comprehension
.59
.00(ns)
.00
.01(ns)
.16
-.04
.05
.00(ns)
.17
.03
.07
Intrinsic goal
.08
.09
R2 = .41
.09
-.07
.58
R2 = .02
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
-.01
.06
Extra-Interaction with Faculty
R2 = .02
-.03
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01(ns)
.01
.07
.01(ns)
.01
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

30. Post- Research Skills Students’ College Goal Orientations
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
Students’ Perceived Learning Outcomes
Now
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Pre-Knowledge & Comprehension
.05
Attention & Interest
-.06
R2 = .21
.18
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
-.01(ns)
R2 = .01
.13
.02
Class Responsibilities
R2 = .08
.64
Pre- Research Skills
.00(ns)
.06
.02
Post- Research Skills
.59
.00(ns)
.00(ns)
.12
-.04
.05
.04
.17
.04
.07
Intrinsic goal
.01(ns)
.10
R2 = .50
.09
-.07
.58
R2 = .02
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
.06
Extra-Interaction with Faculty
R2 = .02
-.03
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01(ns)
.07
.01(ns)
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

31. Post- Communication Skills
Students’ Perceived Learning Outcomes
Now
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Pre-Knowledge & Comprehension
.05
Attention & Interest
R2 = .21
-.06
.18
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
.00
-.01
-.01(ns)
R2 = .01
.13
.02
Class Responsibilities
.00(ns)
R2 = .08
.58
Pre- Research Skills
.00
.12
.59
.02
Post- Communication Skills
.00(ns)
.00
-.05
.10
-.04
.05
.02
.17
.09
.07
Intrinsic goal
.04
.11
R2 = .49
.58
.09
R2 = .02
-.07
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
-.01
.06
Extra-Interaction with Faculty
R2 = .02
-.03
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01
.01(ns)
.07
.01(ns)
.01
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

32. Knowledge & Comprehension
Students’ Perceived Learning Outcomes
Now
Post-
Knowledge & Comprehension
Students’ Perceived Learning Outcomes
at Starting
Students’ College Goal Orientations
Academic Engagement
.33
.16
-.10
.06
.07
.10
.25
.28
.14
-.07
Post- Research Skills
Pre-Knowledge & Comprehension
.05
Attention & Interest
-.06
R2 = .21
.18
Post- Communication Skills
.54
-.11
-.03
Epistemic Goal
.11
.00(ns)
.00
-.01(ns)
-.01
.26
.13
R2 = .01
.02
Class Responsibilities
-.10
.00(ns)
.00
R2 = .08
-.10
Pre- Research Skills
.04
.02
Cumulative GPA
.59
.00(ns)
.00
.11
.01(ns)
-.04
.05
-.08
.17
-.02
.07
Intrinsic goal
.02
.08
R2 = .07
.09
-.07
.58
R2 = .02
Consolidated Learning
-.04
R2 = .18
.14
.11
.04
Pre- Communication Skills
.04
.09
.05
Instrumental Goal
-.01(ns)
-.01
.06
Extra-Interaction with Faculty
-.03
R2 = .02
R2 = .15
.13
Emphasis on Cognitive Strategies in the Major
.01(ns)
.01
.07
.01(ns)
.01
Elaboration
Critical Thinking
Rehearsal
.67
.55
.46

33. NOTE:
Model Fit - Χ2/DF: /39; CFI: .99; RMSEA: .04
Each construct value (factor score) was composed of 1) standardized item scores to address the difference in item-response scale, 2) the mean of item scores weighted by factor loadings, and 3) the resulting scores standardized into a mean of 5 and a standard deviation of 2 .
The SEM includes the error-correlations, but not shown in the figure .
The color of each construct box is correspondent with that of each path coefficient box
The Italic numbers followed by “(ns)” indicate non-significant path coefficients
The numbers following “ “ indicate the strongest impact of each domain on each construct .

34. Conclusion
Structural equation modeling (SEM) yielded several insights into the dimensions that have the greatest impacts on student learning.
Not surprisingly, for any learning outcome, the student’s incoming ability has the greatest effect on current ability.
Following that, the greatest effect on “knowledge and comprehension” is found on the path from “pre-communication skills”, to “intrinsic goal”, to “attention and interest in class”, and from “elaboration”.
The most important factors leading to GPA were found to be the SPLO of “knowledge and comprehension,” and the AE of “class responsibilities”.
“Elaboration” considered a high level of cognitive strategy did not contribute to GPA.
The effect of “critical thinking” on GPA was significant, but negative.
“Elaboration” is the strongest factor of ECSM on all of SPLO.

35. Conclusion, continued There are multiple paths to any desired outcome.
Not all engagement is equal. Engagement activities need to be appropriate to the desired outcome.
Goals are important

36. Contact Us Mark Troy Yunhee Bae To download this presentation, go to:
Yunhee Bae
To download this presentation, go to: