Presentation on theme: "Power up your Program Evaluations Marketing, Development, & Assessment."— Presentation transcript:
Power up your Program Evaluations Marketing, Development, & Assessment
Evaluation Form (paper or online) Responses reviewed by SAO Responses shared with faculty If paper, some reviews placed in the resource center for prospective students to review If online (Zoomerang, SurveyMonkey, etc.) reviews may be downloaded & stored. Other actions taken on a case by case basis. Traditional Approach
Definition: A systematic exploration to discover and judge something that we think has value. Evaluation is Ex Valore in Latin, which means to strengthen or empower Ex – out of, from Valor – strength, value, valor It was originally a means rather than an end A means to empowerment synonymous to positive growth and development A means that involves people, processes, and results associated with what we do What is evaluation?
Practices - Offices that manage the process, third-party subcontracts, host institution services, information sessions, student advising, pre-departure, policies, procedures, etc. Programs - Direct enroll, faculty-led, third-party, consortium, student teaching, internships, etc. (their administration, academics, housing, social, expenses, etc.) Projects - Freshmen orientation, study abroad fair, marketing workshop for faculty, study abroad ambassadors project, welcome back reception, and other special events. to… Prove (show) something is working or needed Improve the practice, program, or project Discover something new for development We can evaluate…
Coordinate evaluation process for multidimensional goals Student Satisfaction Data (regarding Administration, Academics, Housing, Social, Expenses) helps us in marketing and development Learning outcomes, intercultural development, progress towards global citizenship guides our questions regarding assessment Shrewdly and efficiently share information with stakeholders. Power up your evaluations… StakeholdersWhy?Purpose/Goal Prospective Students-Parents, Alumni, potential Donors Earn respect, convince of value-benefits- outcomes, and grow participation Marketing OSA, Offices, Partner Institutions, Organizations, Faculty, & Academic Departments Monitor, grow, improve, foster goals & objectives in programs and operations. Development OSA, the field of study abroad, Faculty, Academic Departments, Accreditation Bodies, Upper Administration, Donors To show that stakes are valid, gain support, lobby, grow the education abroad as a unique & important form of education. Assessment
AbroadScout 2.0 Mission: to empower students, schools, and the education abroad industry with quality information and Web 2.0 tools for marketing, development, and assessment.
AS 2.0 Marketing, Development, Assessment Tools Add your programs and scholarships to the main AS directory (also edit/delete) Create your own PROGRAM FINDER (a customized directory of the sending institutions approved list of programs with scholarships and reviews) Create your own COURSE FINDER (a course articulation database tied to your customized PROGRAM FINDER) - Co managed Customize the AbroadScout Program Evaluation alongside your program partners. One survey per student works for multiple parties (home, provider, host). Create and manage new surveys. Share surveys and results with select partner institutions, offices, etc. Great for development and assessment projects.
AS2.0 Funding Model FREE – All Marketing-Development-Assessment tools. PRO ACCOUNT – Allows you to upload a logo. Cost $300/yr. Marketing - It gives your programs more visibility, as they are randomly sorted among all pro-account programs in the first tier of any search (in the main directory and customized program finders). Also, more information is included in the initial search results. Added Customization - In your Program Finder and Course Finder, it gives your pages a more personalized feel with the logo from your institution on top. ADVERTISING POLICY –To keep information unbiased, and avoid conflicts of interest, the directories (programs, scholarships, and reviews) are completely free of advertising, aside from the pro-account option we offer to all program sponsors equally. FUNDING MODEL – We fund the site through pro-accounts in the directories and banners-links in continent/country sections of the website. These sections contain articles and news, relevant to student interests, many of which are written by students.
Study Abilene Christian University Private Masters Level University 4000 Undergraduate and 700 Graduate 3 Permanent sites, 2 Temp sites, 8 consortium sites Semester and Summer Programs Oxford, England Montevideo, Uruguay Leipzig, Germany
Approx. 220 students abroad split evenly between semester-long and summer programs General Education during semesters Discipline specific during summers ACU On-site directors, ACU visiting faculty, local adjuncts and partner universities Study Abilene Christian University
University Mission and Vision Increase Student Learning Increase participation to reach 50% by 2020 (currently 28%) Expectation that we balance our budget
Accountability to constituents Students, faculty, parents, donors, board of trustees, home and host communities Regional Accrediting Body – SACS Institutional Drivers of Marketing, Development and Assessment Abilene Christian University
What, How and Why Global Perspective Inventory Pre-departure and post completion Embedded in an orientation course Course evaluations Paper and pencil Online From Assessment to Marketing
Online student program evaluation Yes/No and open ended-text box questions obtained at the close of the program Reflection papers embedded in a course Faculty debriefing From Assessment to Marketing
Internal Global Perspectives Inventory (GPI) and online program evaluation results to Program Directors, Senior University Administrators, PR/marketing and Board of Trustees (BOT) Getting from A (Assessment) to M (Marketing)
Course Evaluations – faculty, Dept. Chairs and College Deans, PR/marketing offices Student Reflection papers –faculty, PR/marketing offices Student numbers and demographics - Senior Administrators, BOT – sign leases, purchase property etc… Getting from Assessment to Marketing – Internal (continued)
External Quotes incorporated into Web page articles, ads etc… Information relayed to current and prospective students and their families via promotional and recruiting materials Host community is provided feedback – i.e. volunteer organizations, Property/facility owners, neighbors, partner institutions Getting from A to M (continued)
Increased student learning and self-efficacy Value Added Holistic education Increased student enrollment opportunities for growth Paying the bills Increased funding to improve and expand services and sites C *e*l*e*b*r*a*t*i*o*n*s
Making every one happy vs. creating a culture of quality enhancement and improvement Closing the Loop – Follow up assessment to maximize continual improvement Biting off more than we can chew – trying to do too much C-h-a-l-l-e-n-g-e-s
4-1-4 calendar (5-week January term), abroad and on-campus courses offered About 1,400 UD students abroad in 2010/11; 75% on January programs (1060 in 2011); 90% on short-term programs (summer and winter programs) Over 40% of undergraduates (population 15,000) study abroad About 70 UD programs annually, 50 in January, all faculty led, representing 75% of academic departments About 20 semester/exchange programs (about 130 students total) Study Abroad at Univ. Delaware
Paper evaluation (satisfaction survey) unwieldy to administer data hard to retrieve/summarize/distribute Office of Educational Assessment created 2005 institutional interest in assessing undergraduate learning campus resource for expertise Discovery Learning Experience required for all undergrads as of 2006 good reason for mandatory assessment study abroad counts as DLE! Where We Were
Given that: the number of students and programs is very large; students are enrolled in a wide variety of courses; most are not enrolled in a foreign language course (though many are); programs are located across the globe in both rural and urban settings; the range of program conditions is great (housing, mobility, interaction with host culture); and faculty directors have different program goals: What can we measure that applies to all students? and How can we measure it? Thoughts on How to Move Forward
What We Do – Data Collection Look for general impacts (changes from pre to post) in areas of knowledge, skills, and attitudes (applicable to all programs) Design a short instrument (31 Likert scale items, 2 short answer) Incorporate into required online pre-departure orientation Institute a required post-program assessment (linked to University-wide Discovery Learning Experience Include standard satisfaction survey (subjective feedback on program quality) Use existing online course evaluation system (via registration for UNIV 370, pass/fail, 0- credit, study abroad marker)
What We Do – Data Organization Download satisfaction survey feedback; send as pdf to faculty after grades are submitted Download course feedback; send as pdf to dept. chairs Program coordinators prepare summaries of each program, addressing budget, faculty organization and performance, incidents abroad, student feedback (numerical ratings, selected quotations), and suggestions for next time Summaries sent with course syllabi to dept. chair and dean Assessment results stored and analyzed internally
Largely responsibility of faculty for short-term programs Responses to short-answer questions can be used in publications What We Get and Use: Marketing
What We Get: Development Masses of data: 98% return rate Electronic data collection, retrieval, storage using UDs course evaluation system Student satisfaction survey results helpful for faculty and program administrators Used in program summary sent to faculty, chairs, deans Documentation of problem cases
What We Get: Assessment Quantitative and qualitative pre- and post-sojourn data from about 1400 students on 70 programs annually 98% return rate Electronic data collection, retrieval, storage, using UDs existing online course evaluation system Ongoing data collection Improve academic programming and student learning Friends in Assessment Office Positive contribution to institutional re-accreditation
assessment is self-reported on-campus control group difficult to maintain in-house instrument (untested beyond UD) data backlog (lack of time for analysis) Limitations/Disadvantages summaries very time- consuming and labor- intensive to compose and send not completely objective impact of summaries unknown (Do chairs/ deans read them?)
What kind of assessment or evaluation of education abroad programs is done at your institution? Who sees the results? How are the results used? What prevents results from being used for marketing purposes, for implementing programmatic or procedural changes, or for better understanding student learning? Questions/Discussion
Lisa Chieffo, Ed.D. Associate Director Institute for Global Studies University of Delaware Newark, Delaware, USA How to Find Us Kevin Kehl, Ed.D. Executive Director Center for International Education Abilene Christian University Abilene, Texas, USA Wendy Williamson Director Office of Study Abroad Eastern Illinois University Charleston, Illinois USA