Before & After: What Undergraduates and Alumni Say About Their College Experience and Outcomes Angie L. Miller, NSSE & SNAAP Research Analyst Amber D. Lambert, NSSE & SNAAP Research Analyst Becca Houghton, SNAAP Project Coordinator Assessment Institute October 30 th, 2012

Introduction Surveys are a common means of assessment in higher education Student surveys are conducted on a variety of topics, from student engagement to use of campus resources to faculty evaluations Alumni surveys are used to gather information about satisfaction, acquired skills, and career attainment

Research Questions Question 1: Are there differences in how students and alumni perceive aspects of their institutional experiences and the skills and competencies that they acquire at their institutions? Question 2: What discipline-specific strengths and weaknesses exist when exploring students’ reported development of skills and abilities?

What is SNAAP? Strategic National Arts Alumni Project On-line annual survey of arts graduates Investigates educational experiences and career paths Provides findings to educators and policymakers to improve arts training, inform cultural policy, and support artists

Who is Surveyed? Graduates of: Arts schools, departments, or programs in colleges and universities Independent arts colleges Arts high schools Both graduate and undergraduate degree recipients All arts disciplines

SNAAP Questionnaire Topics 1. Formal education and degrees 2. Institutional experience and satisfaction 3. Postgraduate resources for artists 4. Career 5. Arts engagement 6. Income and debt 7. Demographics

SNAAP 2011 Administration Information Administered in Fall participating institutions 58 postsecondary and 8 high schools Over 36,000 total respondents

What is NSSE? National Survey of Student Engagement NSSE gives a snapshot of college student experiences in and outside of the classroom by surveying first-year and senior students NSSE items represent good practices related to desirable college outcomes Indirect, process measures of student learning and development

NSSE Purpose NSSE annually gathers valid, reliable information on the extent to which students engage in and are exposed to proven educational practices that correspond to desirable learning outcomes. Results indicate how students spend their time and what they gain from college.

NSSE 2012 Administration Information Administered in Spring participating U.S. institutions Over 285,000 total respondents Each year, experimental item sets appended at end of core survey

Question 1: Methodology Participants from 6 institutions that participated in both SNAAP11 and NSSE12 Senior NSSE respondents from arts majors in corresponding SNAAP participating programs (n = 222) Alumni of undergraduate SNAAP programs from graduating cohorts of (n = 593)

Question 1: Respondents CharacteristicsNSSESNAAP Female72%61% U.S. citizen98% First Generation status37%30% Race/Ethnicity* White68%89% Black5%4% Latino/Hispanic7%3% Asian1%5% *SNAAP asks race/ethnicity in a check-all format; NSSE asks race/ethnicity in a forced-choice format

Question 1: SNAAP items

Question 1: NSSE items

Question 1: Results Means comparison for overall rating of institutional experience (4-point scale from “Poor” to “Excellent”) suggests that alumni give higher general appraisals Student Mean Alumni Mean Sig.Effect size (d) Overall experience *.17 *p<.05; **p<.01; ***p<.001

Question 1: Results (cont.) Means comparisons for satisfaction with aspects of time at institution (4-point scale after removing “Not Relevant” option) suggests that alumni give lower specific appraisals for certain aspects Student Mean Alumni Mean Sig.Effect size (d) Academic advising **.22 Career advising ***.39 Opp. for internships ***.31 *p<.05; **p<.01; ***p<.001

Question 1: Results (cont.) Means comparisons for amount of institutional contribution to acquired skills and competencies (4-point scale from “Not at all” to “Very much”) show a similar pattern, with alumni giving lower specific appraisals for certain skills

*p<.05; **p<.01; ***p<.001 Student Mean Alumni Mean Sig.Effect size (d) Research skills **.24 Clear writing ***.33 Persuasive speaking *.20 Project management **.23 Technological skills *.21 Financial & business ***.38 Entrepreneurial **.26 Networking **.27

Question 2: Methodology Participants from 37 institutions that participated in NSSE12 and received the experimental items derived from the SNAAP survey Senior NSSE respondents from all majors* (n = 23,726) *Excluding “other” and “undecided” categories

Question 2: Respondents Characteristics Female64% U.S. citizen95% First Generation status40% Race/Ethnicity White70% Black6% Latino/Hispanic8% Asian5%

Question 2: Respondents’ Majors Primary Major Field Arts8% Humanities9% Biological Sciences12% Business18% Education8% Engineering10% Physical Sciences4% Professional (other)14% Social Sciences16%

Question 2: Results Looking at frequencies for students in the different majors reporting that their institution contributed “Very Much” to acquiring certain skills and competencies reveals interesting patterns Some patterns are expected; others highlight less apparent similarities among major fields

% Reporting Institution Contributed “Very Much” to Acquiring…

Discussion Question 1 Alumni may be viewing their institutional experience as a whole through rose-colored glasses when they think about “the good old days” Post-graduation experiences in the workplace may better enable alumni to reflect on certain aspects of their time Alumni may also learn that they needed to develop some skills more once they have gained work experience

Discussion (cont.) Question 2 The skills that students acquire while at their institutions can vary greatly among different majors Many of these differences are expected (and increase our confidence in the survey instrument) Other patterns link seemingly dissimilar majors, such as arts and engineering, or business and education

Limitations May not represent ALL students and alumni, data only available for those participating in both SNAAP and NSSE (and those receiving experimental NSSE items) Difficulty of surveying alumni: response rates, tracking accurate contact information Relies on self-reported data

Conclusions Important institutional information can be gained through surveying both students and alumni Students may be better able to provide information about affective components of their experience, while alumni may be better judges of specific things needed in the workplace A variety of perspectives can also be gained through exploring differences by majors

March 7-9, 2013 Nashville, TN

Indiana University Center for Postsecondary Research Strategic National Arts Alumni Project National Survey of Student Engagement Angie L. Miller Amber D. Lambert Becca Houghton