For Data Geeks…

Connecting Assessment with Practice: Moving Information from Interesting to Valuable Darlena Jones, Ph.D. Director of Research and Development Educational Benchmarking Inc

Assessment to Practice: A Strategy Assessment Instruments –Defining the goals of an assessment project –Designing assessment instruments –Identifying the various key stakeholders Reporting Strategies –Comprehending the range of reporting methods –Linking effective reporting methods and key institutional stakeholders –Discussing how effective reporting can promote changes in institutional practice Examples in Action –MAP-Works Student Reporting –MAP-Works Faculty/Staff Reporting

Assessment Instruments

Step 1: Understanding the Outcomes of the Assessment Step 3: Use of Assessment Information Step 2: Breadth or Depth of Assessment Step 4: Appropriate Survey Items Keys to Successful Assessment Instruments

Keys to a Successful Assessment Instrument Step 1: What are the outcomes of the assessment? –Program improvement? –Measuring the climate? –Measuring student learning? –Support accreditation / program review?

Keys to a Successful Assessment Instrument Step 2: What is the breadth or depth of your assessment? –To thoroughly educate your audience on a range of items (diagnostic assessment)? Strengths and weaknesses Participant needs Areas of improvement Longitudinal trends Implications and suggestions –To quickly provide “just the facts” (targeted to a specific topic)? Summaries Focus on Outcomes

Keys to a Successful Assessment Instrument Step 3: What will your assessment inform? –Internal to Institution Decision making Program evaluation and improvement Budget allocations Marketing and education Interventions Program development –External to Institution Accreditation reports Grant reports Benchmarking Professional development Publicity for alumni news, local media, etc.

Keys to a Successful Assessment Instrument Step 4: What questions are on your instrument? –Potential Problem: Survey is long and confusing Solution: Create survey that is focused on: –Improvement, not marketing –Performance, not activity (measure outputs, not activity) –But, most important, MISSION –Potential Problem: Survey design is bad; factors have low reliability; results display poor validity Solution: Have “survey experts” design survey or involve people closely connected to issue help design survey –Potential Problem: Don’t know where to begin improvement once analysis is finished Solution: Must have both analyses/systems that help guide action planning

Reporting Strategies

Step 1: Understanding the Audience Step 3: Using Valuable Information Step 2: Appropriate Form for Information Step 4: Choosing the Best Delivery Method Keys to Successful Reporting

Understanding the Audience Step 1: Who will be reading your assessment report? –Internal Stakeholders Administrative decision-makers Boards of trustees Budget administrators Faculty or staff Internal governing bodies (Faculty/Staff Senate, Unions, etc.) Students –External Stakeholders Accreditation bodies & reviewers Alumni Community members Donors Grant reviewers Prospective students & parents State and federal governments

Understanding the Audience Step 1 (cnt): What is your audience’s experience with information? –Quantitative vs. Qualitative How comfortable are they with statistics? Do they prefer narratives or numbers? –Interest and experience - How much explanation regarding the… Topic? Assessment methods? Results? Implications? –Time available - How much time can or will they spend reading the results? –Level - Will they use university level data? College level? Department level? Individual level?

Appropriate Form for Information Step 2: What is the best form for the information? –Types of reports Executive summaries Comprehensive reports Assessment summaries Notes, brochures, flyers, and memos Institutional snapshots Interactive data –Easily read? What does the report look like (size of font, appearance, visuals, etc.)? Do they want to read this? Does it draw them in? Does it intimidate them or overwhelm them?

Using Valuable Information Step 3: What is the content of the report? –Importance: Does this report… Include important issues? Highlight and emphasize the important results? Differentiate between important and non-important results? –Usefulness: Does this report… Discuss the implications of the results? Clearly link the results to practice? Help practitioners determine what should be done? Differentiate between useful results and interesting results?

Choosing the Best Delivery Method Step 4: How will you deliver the assessment results? –Media formats Paper Electronic (websites, downloadable files, CDs, s, etc.) Oral presentations Combinations –Easily accessible? How hard is the information for Stakeholders to access or find? Can they find what they need quickly?

Examples in Action: MAP-Works Student Report EBI’s “best practices” model of reporting

Who is Responsible? Who is responsible for student success on your campus? –Enrollment Management/ Retention? –Student Affairs? –Academic Affairs? What information do you know about this first-year student? Student ID: YD HS GPA: 3.93 SAT Verbal: 29 Location: In state Gender: Female Race: African American Age: 18 Major: Undecided Do you really know them? Enrollment Management/R etention Student Affairs Academic Affairs

Paradigm Shift What would happen if… –ALL faculty/staff were responsible for student success? –YOU knew student was struggling? Could you do something about it before it was too late? Enrollment Management / Retention Residence Hall Staff Academic Advisor First-Year Seminar Instructor Academic Department Heads Financial Aid Minority Student Affairs I’m really homesick I don’t think I can afford college My roommate and I argue all the time Student Affairs Academic Affairs I’m thinking about transferring I’m struggling in my math class

MAP-Works Mission – 4 Way Academic Success: Improve students' ability to succeed academically by realigning behavior with grade expectations and focusing on elements of academic success Retention: Minimize percentage of capable students who drop out due to issues that could have been addressed by self- awareness or timely intervention by staff/faculty Student Development: Facilitate the establishment of relationships, address homesickness, identify residence hall living issues Student Involvement: Connect students with campus resources to facilitate involvement with student organizations and campus programming

MAP-Works History 1988, Ball State developed concept 2005, Ball State partnered with EBI to create MAP-Works 1989 to 2004, Ball State used MAP in-house Fall approximately 40 campuses will use MAP- Works! In 1988, Ball State had a number of concerns… –First-year students arrived with unrealistic expectations (academics, grades, housing, etc.) –Retention rates were not as high as they wanted them to be –Faculty and staff were concerned that identifying student issues at mid-term was too late –Faculty and staff wanted better data about incoming students

MAP-Works Process Expectations Behaviors Social Norming Expectations Campus Resources Student Summary Scan Students Student Profile Institution Profile Campus Resources

Understanding the Student’s Experience Survey and Profile Items Academic Integration –Academic Self-Efficacy –Core Academic Behaviors –Advanced Academic Behaviors –Commitment to Higher Education Academic Integration –Academic Self-Efficacy –Core Academic Behaviors –Advanced Academic Behaviors –Commitment to Higher Education Self-Assessment –Communication Skills –Analytical Skills –Personal Management –Time Management –Health and Wellness –Potential Issues (stress, financial, etc.) Self-Assessment –Communication Skills –Analytical Skills –Personal Management –Time Management –Health and Wellness –Potential Issues (stress, financial, etc.) Profile Information –Student information like gender and race/ethnicity –Entrance exam scores –# credit hours enrolled Profile Information –Student information like gender and race/ethnicity –Entrance exam scores –# credit hours enrolled Social Integration –Campus Relationships –Living Environment (on/off campus) –Roommate Relationships –Homesickness Social Integration –Campus Relationships –Living Environment (on/off campus) –Roommate Relationships –Homesickness

Who benefits from MAP-Works? –First-Year Students –Departments like… Housing & Residence Life Academic Advising Enrollment Management / Retention First-Year Seminar Instructors Minority Student Affairs Athletic Department Student Activities –Academic Assessment & Institutional Research –Upper Administration Who Benefits from MAP-Works?

MAP-Works Student Reports Student Reporting –Individualized on-line report provided directly to students within days of assessment 3 Main Reporting Purposes –Purpose 1: Realign expectations –Purpose 2: Information to help them plan for their success –Purpose 3: Connect with appropriate campus resources

Purpose 1: Realign Expectations (Example 1) Report provides benchmark information to help students realign expectations REPORTING TIP: Only provide the most valuable information in an summary report. Keep the “interesting” information for the larger report.

Purpose 1: Realign Expectations (Example 2) Report provides benchmark information to help students realign expectations REPORTING TIP: Know your audience! Consider using very simple charts to relay complex information (most students don’t have experience reading complex charts/tables).

Purpose 2: Plan for Success Report provides feedback to help student understand need to plan for their future success REPORTING TIP: Consider providing written explanations to help the reader draw conclusions.

Purpose 3: Connect with Resources School lists campus resources/offices that link with reporting area REPORTING TIP: Consider providing additional information the reader may want to access.

Evaluating MAP-Works Student Reporting Step 1: Understanding the Audience – Students have less experience with information; reporting is easy to read Step 3: Using Valuable Information - Targeted to helping them be successful in school Step 2: Appropriate Form for Information – Extremely visual and colorful. Minimal information per slide. Step 4: Choosing the Best Delivery Method – Web based is a delivery method most expected by this generation Keys to Successful Reporting

Examples in Action: MAP-Works Faculty/Staff Report

MAP-Works Faculty/Staff Reporting Interactive system for faculty and staff –Residence hall staff –Academic advisors –First-Year Seminar Instructors –Other staff positions? 3 Main Reporting Purposes –Purpose 1: Identify students who may benefit from personalized attention –Purpose 2: Provide information for one-on-one meetings with students –Purpose 3: Provide input regarding programming and training needs

Purpose 1: Identifying Students (Example 1) These students are not adjusting academically or socially and are not committed to their institution. REPORTING TIP: Consider creating high level “dashboards” that provides a lot of information in an easy-to-read format.

Purpose 1: Identifying Students (Example 2) These students very interested in leadership positions – Recruit them! These students are moderately interested in a leadership position – Convince them? These students definitely not interested in a leadership position – Why? Important on résumé REPORTING TIP: Consider using simple color codes instead of statistical information to relay information.

Purpose 2: Individual Meetings (Example 1) Kimberly is having issues adjusting to college (both academically and socially) and early warning indicators are not good. REPORTING TIP: Consider creating “dashboard” reports that give a 30,000 foot look at the data.

Purpose 2: Individual Meetings (Example 2) REPORTING TIP: Consider providing easy access to in-depth information if the reader needs more clarification.

Purpose 3: Programming These students have self rated themselves low in Public Speaking skills. How could this be addressed through programming? REPORTING TIP: Consider that the information you provide can be used in multiple ways and how that information can be presented to best tell the story.

Evaluating MAP-Works Faculty/Staff Reporting Step 1: Understanding the Audience – Reporting is color-coded for readability; No special training required Step 3: Using Valuable Information – Highlights students with more critical issues; dashboards are focused on critical transition issues Step 2: Appropriate Form for Information – Extremely visual and colorful. Easy navigation. Step 4: Choosing the Best Delivery Method – Web based is a delivery method that provides most flexibility in searching through information Keys to Successful Reporting

Additional Questions and Discussion… Darlena Jones, Ph.D. Director of Research and Development Educational Benchmarking Inc For more information about MAP-Works, contact Todd Pica, Or, visit