Assessing SAGES with NSSE data Office of Institutional Research September 25 th, 2007

Introduction Today’s talk presents an analysis of the impact of SAGES participation on a number of NSSE survey items Focus on first-year responses only as no full- implementation classes have graduated Report will first focus on NSSE benchmark scales followed by a discussion of individual NSSE items related to SAGES learning goals

Analysis The seven years of NSSE data collected by Case has been divided into four groups: ◦ Pre-SAGES (2000, 2001) ◦ Students in the SAGES pilot ( ) ◦ Students not in the SAGES pilot ( ) ◦ Full-Implementation Classes ( ) An analysis of covariance (ANCOVA) was conducted to determine group differences on NSSE benchmarks and survey items All analyses control for first-year reported major

NSSE Benchmarks In order to condense NSSE’s 80+ survey items into easily discussed and analyzed scales, NSSE has developed five conceptually and statistically valid “benchmark” scales. Scales include “Active and Collaborative Learning,” “Academic Challenge,” “Student-Faculty Interactions,” and “Supportive Campus Environment.” In 2005 NSSE changed the calculation of the fifth benchmark, “Enriching Educational Experiences,” making pre-2005 longitudinal comparisons of this scale unreliable.

Active and Collaborative Learning Measures the extent to which students engage in classroom activities and collaborate with others to solve problems. Items include: asked questions in class, made class presentations, worked with other students on projects during class, worked with other students outside of class.

Active and Collaborative Learning Results indicated that those in the SAGES pilot had significantly higher scores than all other students. Additionally, those in the full-implementation of SAGES had, on average, significantly higher scores than pre- SAGES students.

Active and Collaborative Learning Results indicated that those in the SAGES pilot had significantly higher scores than all other students. Additionally, those in the full-implementation of SAGES had significantly higher scores than pre-SAGES students.

Academic Challenge Measures the extent to which students exert—and institutions demand— academic effort. Items include: time spent preparing for class, number of assigned textbooks, and the extent to which the campus environment is perceived to emphasize academics.

Academic Challenge Results revealed no group differences on the Academic Challenge measure

Student-Faculty Interaction Measures the extent to which students learn and solve problems by interacting with faculty members. Items include: the extent to which students discussed grades with faculty, worked with faculty members on activities other than coursework, and discussed career plans with faculty.

Student-Faculty Interaction Results indicated that those in the SAGES pilot had significantly higher scores than those in the full- implementation class There were no significant differences among the remaining three groups

Student-Faculty Interaction Results indicated that those in the SAGES pilot had significantly higher scores than those in the full- implementation classes There were no significant differences among the remaining three groups

Supportive Campus Environment Measures the extent to which students believe that the institution is committed to their success and cultivates positive relations among different groups on campus. Items include: quality of relationships with faculty, quality of relationships with peers, and the extent to which the campus environment is perceived to provide support to succeed socially and academically.

Supportive Campus Environment Results indicated that those in the SAGES pilot had significantly higher scores than pre-SAGES students and students in the full-implementation classes There were no significant differences among the remaining three groups

Supportive Campus Environment Results indicated that those in the SAGES pilot had significantly higher scores than pre-SAGES students and students in the full-implementation classes There were no significant differences among the remaining three groups

Individual Items SAGES goals focus on classroom participation, developing writing and speaking skills, and academic advising. The following slides examine 6 individual NSSE items relevant to these goals: ◦ Asked questions in class ◦ Gave a class presentation ◦ My experience at Case has contributed to my ability to write clearly and effectively ◦ My experience at Case has contributed to my ability to speak clearly and effectively ◦ Satisfaction with University administration ◦ Satisfaction with academic advising

Individual Items For both of these items, those in the SAGES pilot outperformed all other groups; however, students in the full-implementation classes had significantly higher scores than pre-SAGES students.

Individual Items For both of these items, there were no differences between those in the SAGES pilot and those in the full-implementation classes. Those in the SAGES pilot and the full-implementation classes outperformed the other two groups.

Individual Items For “Satisfaction with Advising” there were no year-to-year differences. For “Satisfaction with Administration,” those in the SAGES pilot had significantly higher scores than all other groups. Those in the full- implementation classes had significantly lower scores than all other groups

Conclusions – Benchmarks For three of the four benchmark scales, students in the SAGES pilot significantly outperformed at least one other group. Of the four benchmark scales, only one— Active and Collaborative Learning— significantly increased from pre-SAGES to the full-implementation class.

Conclusions – Individual items Students in the full-implementation classes scored significantly higher than pre-SAGES students on a number of items: ◦ Asked questions in class ◦ Gave a class presentation ◦ My experience at Case has contributed to my ability to write clearly and effectively ◦ My experience at Case has contributed to my ability to speak clearly and effectively There were no group differences on ratings of satisfaction with advising There was a significant drop in ratings of satisfaction with administration from pre-SAGES to the full-implementation classes.

Limitations NSSE is a survey of the entire experience at Case, not an assessment of SAGES. These analyses statistically control for first- year expected major only. Significant differences from pre- to post-full- implementation of SAGES can be inferred to be due in part to the change in curriculum, but may also be due to unmeasured—or un- measurable—influences.

Thank You! Questions? Concerns? Contact: Tom Geaghan