Are those Rose-Colored Glasses You are Wearing?: Student and Alumni Survey Responses Amber D. Lambert, Ph.D. Angie L. Miller, Ph.D. Association for the Study of Higher Education 38 th Annual Conference November 16 th, 2013

Introduction & Literature Review Surveys are a common means of assessment in higher education (Kuh & Ikenberry, 2009) Student surveys are conducted on a variety of topics, from student engagement to use of campus resources to faculty evaluations (Kuh & Ewell, 2010) Alumni surveys are used to gather information about satisfaction, acquired skills, and career attainment (Cabrera, Weerts, & Zulick, 2005)

Introduction & Literature Review Institutions claim to prepare students with a multitude of transferable skills in addition to pure content knowledge from major (Tait & Godfrey, 1999) Effective communication Analytical & creative thinking AAC&U has recently addressed many of these types of skills as essential learning outcomes for higher education Mastery of these skills should lead to success in the workplace (Stasz, 2001)

Research Questions Question 1: Are there differences in how students and alumni perceive aspects of their institutional experiences and the skills and competencies that they acquire at their institutions? Question 2: Do alumni evaluate their institutions with rose-colored glasses, or do they evaluate their education more harshly once they gain a more practical knowledge of the working world? Question 3: Finally, if differences between students and alumni do exist, whose account should be given precedence?

Methodology This study uses data from the 2011 Strategic National Arts Alumni Project (SNAAP) and the 2012 National Survey of Student Engagement (NSSE) What is SNAAP? On-line annual survey of arts graduates Investigates educational experiences and career paths Provides findings to educators and policymakers to improve arts training, inform cultural policy, and support artists

Who does SNAAP Survey? Graduates of: Arts schools, departments, or programs in colleges and universities Independent arts colleges Arts high schools Both graduate and undergraduate degree recipients All arts disciplines

SNAAP Questionnaire Topics 1. Formal education and degrees 2. Institutional experience and satisfaction 3. Postgraduate resources for artists 4. Career 5. Arts engagement 6. Income and debt 7. Demographics

SNAAP 2011 Administration Information Administered in Fall participating institutions 58 postsecondary and 8 high schools Over 36,000 total respondents

What is NSSE? National Survey of Student Engagement NSSE gives a snapshot of college student experiences in and outside of the classroom by surveying first-year and senior students NSSE items represent good practices related to desirable college outcomes Indirect, process measures of student learning and development

NSSE 2012 Administration Information Administered in Spring participating U.S. institutions Over 285,000 total respondents Each year, experimental item sets appended at end of core survey

Methodology: Sample For this study, participants came from 6 institutions that participated in both SNAAP11 and NSSE12 administrations Senior NSSE respondents from arts majors in corresponding SNAAP participating programs (n = 222) Alumni of undergraduate SNAAP programs from graduating cohorts of (n = 593)

Variables: SNAAP items

Variables: NSSE items

Data Analysis Analysis of covariance (ANCOVA) was conducted to determine whether differences of reported satisfaction and skill development exist between graduating seniors and alumni Control variables included gender, race, U.S. citizenship status, and first-generation status Adjusted means and statistical significance Cohen’s d as measure of effect size

Results Adjusted means comparison for overall rating of institutional experience (4-point scale from “Poor” to “Excellent”) suggests that alumni give higher general appraisals Student Mean Alumni Mean Sig.Effect size (d) Overall experience *.17 *p<.05; **p<.01; ***p<.001

Results (cont.) Adjusted means comparisons for satisfaction with aspects of time at institution (4-point scale after removing “Not Relevant” option) suggest that alumni give lower specific appraisals for certain aspects Student Mean Alumni Mean Sig.Effect size (d) Academic advising *-.16 Career advising ***-.35 Opp. for internships **-.27 *p<.05; **p<.01; ***p<.001

Results (cont.) Adjusted means comparisons for amount of institutional contribution to acquired skills and competencies (4-point scale from “Not at all” to “Very much”) show a similar pattern, with alumni giving lower specific appraisals for certain skills

*p<.05; **p<.01; ***p<.001 Student Mean Alumni Mean Sig.Effect size (d) Research skills **-.23 Clear writing ***-.30 Persuasive speaking *-.21 Project management *-.21 Financial & business ***-.38 Entrepreneurial **-.27 Leadership skills *-.21 Networking ***-.28

Discussion Alumni may be viewing their institutional experience as a whole through rose-colored glasses when they think about “the good old days” However, when considering more nuanced aspects of their educational experiences, alumni perceptions may have a more lackluster pallor

Discussion Post-graduation experiences in the workplace may better enable alumni to reflect on certain aspects of their time Alumni were less satisfied than graduating seniors in the areas of academic advising, career advising, and opportunities for internships or degree-related work May be the case that as students, respondents do not realize the need for better advising or internships until they enter the workforce

Discussion Alumni may also learn that they needed to develop some skills more once they have gained work experience Writing, speaking, networking, and leadership are important aspects of communication that may be experienced differently in an applied setting, such as the workplace, in comparison to a classroom situation Some task-based procedural skills like research, project management, finance, and entrepreneurship may also be more completely understood and valued once an individual transitions from student to employee Also possible that once alumni enter the workforce, they reference skill levels in comparison with colleagues who are quite advanced in these skills after years (or decades) of actual use

Limitations Effect sizes small in magnitude May not represent ALL students and alumni, data only available for those participating in both SNAAP and NSSE (and those receiving experimental NSSE items) Relies on self-reported data However, most studies looking at student self- reports in higher education suggest that self- reports and actual abilities are positively related (Anaya, 1999; Hayek et al., 2002; Laing et al., 1988; Pace, 1985; Pike, 1995).

Conclusions Important institutional information can be gained through surveying both students and alumni Students may be better able to provide information about affective components of their experience, while alumni may be better judges of specific things needed in the workplace Being closer in time to the experience may have advantage in terms of memory accuracy, but temporal distance may have advantage of reflective insight

References Anaya, G. (1999). College impact on student learning: Comparing the use of self-reported gains, standardized test scores, and college grades. Research in Higher Education, 40, Cabrera, A.F., Weerts, D.J., & Zulick, B.J. (2005). Making an impact with alumni surveys. New Directions for Institutional Research, 2005: doi: /ir.144 Hayek, J. C., Carini, R. M., O’Day, P. T., & Kuh, G. D. (2002). Triumph or tragedy: Comparing student engagement levels of members of Greek-letter organizations and other students. Journal of College Student Development, 43(5), Kuh, G. D. & Ewell, P. T. (2010). The state of learning outcomes assessment in the United States. Higher Education Management and Policy, 22(1), Kuh, G. D. & Ikenberry, S. O. (2009). More than you think, less than we need: Learning outcomes assessment in American higher education, Urbana, IL: University of Illinois and Indiana University, National Institute of Learning Outcomes Assessment. Laing, J., Swayer, R., & Noble, J. (1989). Accuracy of self-reported activities and accomplishments of college-bound seniors. Journal of College Student Development, 29(4), Pace, C. R. (1985). The credibility of student self-reports. Los Angeles: The Center for the Study of Evaluation, Graduate School of Education, University of California at Los Angeles. Pike, G. R. (1995). The relationship between self-reports of college experiences and achievement test scores. Research in Higher Education, 36(1), Stasz, C. (2001). Assessing skills for work: Two perspectives. Oxford Economic Papers, 3, Tait, H., & Godfrey, H. (1999). Defining and assessing competence in generic skills. Quality in Higher Education, 5(3),

Questions or Comments? Contact Information: Amber D. Lambert Angie L. Miller Strategic National Arts Alumni Project National Survey of Student Engagement