1. Before & After: What Undergraduates and Alumni Say About Their College Experience and Outcomes Angie L. Miller, NSSE & SNAAP Research Analyst Amber D. Lambert, NSSE & SNAAP Research Analyst Becca Houghton, SNAAP Project Coordinator Assessment Institute October 30 th, 2012

2. Introduction Surveys are a common means of assessment in higher education Student surveys are conducted on a variety of topics, from student engagement to use of campus resources to faculty evaluations Alumni surveys are used to gather information about satisfaction, acquired skills, and career attainment

3. Research Questions Question 1: Are there differences in how students and alumni perceive aspects of their institutional experiences and the skills and competencies that they acquire at their institutions? Question 2: What discipline-specific strengths and weaknesses exist when exploring students’ reported development of skills and abilities?

4. What is SNAAP? Strategic National Arts Alumni Project On-line annual survey of arts graduates Investigates educational experiences and career paths Provides findings to educators and policymakers to improve arts training, inform cultural policy, and support artists

5. Who is Surveyed? Graduates of: Arts schools, departments, or programs in colleges and universities Independent arts colleges Arts high schools Both graduate and undergraduate degree recipients All arts disciplines

6. SNAAP Questionnaire Topics 1. Formal education and degrees 2. Institutional experience and satisfaction 3. Postgraduate resources for artists 4. Career 5. Arts engagement 6. Income and debt 7. Demographics

7. SNAAP 2011 Administration Information Administered in Fall 2011 66 participating institutions 58 postsecondary and 8 high schools Over 36,000 total respondents

8. What is NSSE? National Survey of Student Engagement NSSE gives a snapshot of college student experiences in and outside of the classroom by surveying first-year and senior students NSSE items represent good practices related to desirable college outcomes Indirect, process measures of student learning and development

9. NSSE Purpose NSSE annually gathers valid, reliable information on the extent to which students engage in and are exposed to proven educational practices that correspond to desirable learning outcomes. Results indicate how students spend their time and what they gain from college.

10. NSSE 2012 Administration Information Administered in Spring 2012 546 participating U.S. institutions Over 285,000 total respondents Each year, experimental item sets appended at end of core survey

11. Question 1: Methodology Participants from 6 institutions that participated in both SNAAP11 and NSSE12 Senior NSSE respondents from arts majors in corresponding SNAAP participating programs (n = 222) Alumni of undergraduate SNAAP programs from graduating cohorts of 2001-2010 (n = 593)

12. Question 1: Respondents CharacteristicsNSSESNAAP Female72%61% U.S. citizen98% First Generation status37%30% Race/Ethnicity* White68%89% Black5%4% Latino/Hispanic7%3% Asian1%5% *SNAAP asks race/ethnicity in a check-all format; NSSE asks race/ethnicity in a forced-choice format

13. Question 1: SNAAP items

15. Question 1: NSSE items

17. Question 1: Results Means comparison for overall rating of institutional experience (4-point scale from “Poor” to “Excellent”) suggests that alumni give higher general appraisals Student Mean Alumni Mean Sig.Effect size (d) Overall experience3.273.38*.17 *p<.05; **p<.01; ***p<.001

18. Question 1: Results (cont.) Means comparisons for satisfaction with aspects of time at institution (4-point scale after removing “Not Relevant” option) suggests that alumni give lower specific appraisals for certain aspects Student Mean Alumni Mean Sig.Effect size (d) Academic advising3.002.79**.22 Career advising2.792.40***.39 Opp. for internships2.682.36***.31 *p<.05; **p<.01; ***p<.001

19. Question 1: Results (cont.) Means comparisons for amount of institutional contribution to acquired skills and competencies (4-point scale from “Not at all” to “Very much”) show a similar pattern, with alumni giving lower specific appraisals for certain skills

20. *p<.05; **p<.01; ***p<.001 Student Mean Alumni Mean Sig.Effect size (d) Research skills3.293.10**.24 Clear writing3.212.95***.33 Persuasive speaking2.962.79*.20 Project management3.233.03**.23 Technological skills3.253.08*.21 Financial & business2.241.92***.38 Entrepreneurial2.232.00**.26 Networking3.072.84**.27

21. Question 2: Methodology Participants from 37 institutions that participated in NSSE12 and received the experimental items derived from the SNAAP survey Senior NSSE respondents from all majors* (n = 23,726) *Excluding “other” and “undecided” categories

22. Question 2: Respondents Characteristics Female64% U.S. citizen95% First Generation status40% Race/Ethnicity White70% Black6% Latino/Hispanic8% Asian5%

23. Question 2: Respondents’ Majors Primary Major Field Arts8% Humanities9% Biological Sciences12% Business18% Education8% Engineering10% Physical Sciences4% Professional (other)14% Social Sciences16%

24. Question 2: Results Looking at frequencies for students in the different majors reporting that their institution contributed “Very Much” to acquiring certain skills and competencies reveals interesting patterns Some patterns are expected; others highlight less apparent similarities among major fields

25. % Reporting Institution Contributed “Very Much” to Acquiring…

33. Discussion Question 1 Alumni may be viewing their institutional experience as a whole through rose-colored glasses when they think about “the good old days” Post-graduation experiences in the workplace may better enable alumni to reflect on certain aspects of their time Alumni may also learn that they needed to develop some skills more once they have gained work experience

34. Discussion (cont.) Question 2 The skills that students acquire while at their institutions can vary greatly among different majors Many of these differences are expected (and increase our confidence in the survey instrument) Other patterns link seemingly dissimilar majors, such as arts and engineering, or business and education

35. Limitations May not represent ALL students and alumni, data only available for those participating in both SNAAP and NSSE (and those receiving experimental NSSE items) Difficulty of surveying alumni: response rates, tracking accurate contact information Relies on self-reported data

36. Conclusions Important institutional information can be gained through surveying both students and alumni Students may be better able to provide information about affective components of their experience, while alumni may be better judges of specific things needed in the workplace A variety of perspectives can also be gained through exploring differences by majors

37. http://3millionstories.com/ March 7-9, 2013 Nashville, TN

38. Indiana University Center for Postsecondary Research Strategic National Arts Alumni Project snaap@indiana.edusnaap@indiana.edusnaap.indiana.edu National Survey of Student Engagement nsse@indiana.edunsse@indiana.edunsse.indiana.edu Angie L. Miller anglmill@indiana.eduanglmill@indiana.edu Amber D. Lambert adlamber@indiana.eduadlamber@indiana.edu Becca Houghton rfagen@indiana.edurfagen@indiana.edu