Office of Institutional Research Tracking Alumni Outcomes: A measure of educational effectiveness K. Tracy Barnes

Standard 8: Educational Effectiveness Based on verifiable information, the institution understands what its students have gained as a result of their education and has useful evidence about the success of its recent graduates. This information is used for planning and improvement, resource allocation and to inform the public about the institution. Student achievement is at a level appropriate for the degree awarded. Office of Institutional Research

Background We convened a group of interested stakeholders from offices across campus Settled on the minimum amount of information we deemed essential and decided to use several different ways to gather information on alumni 5 and 10 years post graduation. Worried less about “response rate” and more about “knowledge rate” – on what percentage of alumni can we get some information? Office of Institutional Research

Goals for today Describe the results of our effort to date. Will use graduate alumni (PhD and MA) to illustrate all the steps we take to gather the information but it applies to students at other levels Describe relative accuracy of sources, including LinkedIn Show how results are shared Describe next steps Office of Institutional Research

Pilot Project: 2008 Graduate Alumni In 2013 conducted a pilot for alumni who had graduated in 2008 and 2003 Collected information from multiple sources, departments, alumni relations, survey and LinkedIn Compared information across the sources Today will discuss the graduate school alumni from the class of 2008 Office of Institutional Research

Comprehensiveness of Sources Which source has any (not necessarily current) employment information? Office of Institutional Research Class of 2008 (n=369) Alumni Relations (possibly outdated)120 (33%) Department (possibly outdated)240 (65%) Survey160 (43%) Total from All Sources273 (74%)

Added Value of LinkedIn Office of Institutional Research Class of 2008 (n=369) Alumni Relations (possibly outdated)120 (33%) Department (possibly outdated)240 (65%) Survey160 (43%) LinkedIn129 (35%) Total from All Sources311 (84%)

Accuracy of Sources Survey is standard: 100% accurate LinkedIn is more accurate than existing sources Office of Institutional Research Job info from both survey and… NSources Agree Sources Conflict Accuracy Alumni Relations % Department % LinkedIn %

Repeated in 2014 Only surveyed alumni without updated (within 6 months) info on file –Response rate: 43% Only checked LinkedIn for those without updated info or survey response –Exception: checked small sample against survey, again found around 90% accuracy Office of Institutional Research

Added Value of LinkedIn Comparable to prior year Office of Institutional Research Class of 2008 (n=369) Class of 2009 (n=465) Alumni Relations (possibly outdated) 120 (33%)268 (58%) Department (possibly outdated)240 (65%)178 (38%) Survey160 (43%)80 (17%) LinkedIn129 (35%)140 (30%) Total from All Sources311 (84%)370 (80%)

Current Information If you only use information that can be verified as within the last 6 months Office of Institutional Research Class of 2009 (n=465) Alumni Relations81 (17%) Department0 (0%) Survey80 (17%) LinkedIn140 (30%) Total from All Sources292 (63%)

Graduate Alumni Knowledge Rate Office of Institutional Research Class of 2008 unverified Class of 2009 unverified Class of 2009 verified Alumni Relations120 (33%)268 (58%)81 (17%) Department240 (65%)178 (38%)0 (0%) Survey160 (43%)80 (17%) LinkedIn129 (35%)140 (30%) Total from All Sources 311 (84%)370 (80%)292 (63%)

LinkedIn Process and Logistics Easier than we had expected 1 minute to look up each alum Second time around: only surveyed alumni for whom we did not have complete and recent information from Alumni Relations or department Office of Institutional Research

LinkedIn Instructions for Intern Who: alums who didn’t respond to the survey and haven’t updated info in past 6 months Make sure info is not out of date (e.g., most recent employment pre- dates graduation date) Caution: Verify that the LinkedIn profile actually belongs to alum. Matching on name is not sufficient. Did the person graduate from Brown? Is the degree in the correct field? Is the LinkedIn graduation year within one year of actual graduation year? Note possible name change Office of Institutional Research

Disseminating the Information Developed a dashboard with the graduate alumni data research/surveys/graduate-alumni research/surveys/graduate-alumni Exploring ways to publish undergraduate outcome data (e.g, Focal Point) Office of Institutional Research

Conclusions Supplementing department and AR information with LinkedIn and survey provides reasonably high “knowledge” rate Graduate School now has a link from each department to the website. We know anecdotally that Focal Point is used by advisors, students (prospective and enrolled) and their parents We feel this information is an important piece of assessing our educational “effectiveness” Office of Institutional Research

Future Plans Currently: complete a similar process for the undergraduate alumni (look up in LinkedIn, compare to what we know from the survey and other sources) –From most recent (2015) survey we compared 50 undergraduate alumni survey responses to LinkedIn. 39 respondents (78%) had LinkedIn profiles and for 90% employment information matched Will use the National Student Clearinghouse to determine how many have gone on for further education Will modify the graduate alumni display to use multiple years of data, combining all three into a 1,5,10 year post graduation category Will add the five-year out undergraduate alumni information to focal point. Office of Institutional Research