Presentation on theme: "Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002."— Presentation transcript:
Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002
The Data Audit Toolkit n Developed Through a Partnership >The Policy Center on the First Year of College John Gardner, Betsy Barefoot, Randy Swing, Mike Siegel, and Marc Cutright >NCHEMS Karen Paulson and Peter Ewell n Generously Supported by: >The Pew Charitable Trusts >The Atlantic Philanthropies
Pilot Institutions n Augustana College (IL) n Blue Ridge Community College (VA) n Lynchburg College (VA) n Northeast State Technical Community College (TN) n Ohio University n Santa Fe Community College (FL) n The University of Texas at El Paso n University of Cincinnati n University of Minnesota Duluth n Washington State University
Data Audit The process of identifying data resources and uses wherever they may be within an institution and gathering them into a useable information system. NCHEMS Administrative Rationale manual, page 15
Basic Premise I n Most of the Data Needed for Meaningful Analysis Already Reside in Institutional Data Systems >Admissions >Registration/Transcripts >Student Surveys >Assessment Databases >Individual Service Office Records
Basic Premise II n The Major Task Is: >To Organize These Data in Useful Ways >To Make Access and Analysis as Flexible and Straightforward as Possible
The Data Audit Process n Is NOT Itself an End n Is a Means for Understanding and Improving the Use of Data and Information on Campus Key to Improving the First Year of College
…we tend to assume that all first-year programs were implemented as planned and that the experiences of all students were uniform.
In Reality There Are Three First-Year Experiences* n The Official One (Designed and Published Plans) n The Delivered One (Actual Institutional Actions) n The Experienced One (Reality Including Student Choice) * Adapted from Joan Stark
The Main Question…. n How can your institution analytically disentangle the many elements of the first college year and provide evidence about the effectiveness, or lack thereof, of related first-year programs, policies, and procedures?
Some Preliminary Questions n What Is Unique About the First Year of College at Your Institution? n What Is it Like to Walk in a Students Shoes at Your Institution? n How Many Surveys, Tests, or Assessments Does a Student Really Experience During the Year? n Does Any Information from These Data Collections Get Communicated Back to the Students? How? n How Are These First-Year Data Used by Faculty and Administrators?
A Quick Self-Assessment n What Are the Primary Sources of Data About Students, Curricula, and ProgramsSpecifically Related to the First Year of CollegeCurrently in Place at Your Institution? n Are Any of These Underutilized? Why? n Whats Missing About Which You Would Like Information?
Why a Data Audit? n Identify and Inventory Data Sources n Identify and Inventory Data Needs n Support Assessment of... >What Happened >What Mattered n Foster a Culture of Data Use
Primary Data Audit Activities The Supply Side Conducting a Campus-Wide Examination of Existing Data Sources The Demand Side Determining Which Data Are Most Needed for Evaluation, Assessment, and Decisionmaking
Key Components of a Data Audit n Identifying Sources of Data n Inventorying These Data Sources n Compiling Information About Data Sources n Identifying Gaps in Data Sources n Assessing What Data Users Need and Want to Know n Determining Which Analyses (Existing or Suggested) Have Utility for Stakeholders of Your Institution or Unit
Organizing a Data Audit n Institution-Wide Committee >IR, Student Affairs, Academic Affairs, Advisors, Orientation Staff, Faculty, Registrar Staff, Students, etc. n Team Members >People Directly Involved with the Institution >People Directly Involved with Data >First-Year Professionals not Normally Data Connected >Leader(s) with Broad Campus Support and Respect The Data Audit Leadership Team
The Right Attitude n Fresh Perspective n Open Attitude n Collaborative Approach Not a Gotcha Mentality
Visiting Potential Data Sites n Reasons for Physical Visits >Honors Unit Personnel on Their Turf >Builds Relationships >Allows You to Read Reactions >Allow You to Do Immediate Follow-Up and Collect Artifacts >Discovers Hidden Databases
Types of Data n Official Data >Registrar Systems >Admissions >Institutional Research Offices >Assessment Offices n Unofficial (Guerrilla) Databases >Unit Records Kept in Local Computers >Viewed as Single Purpose/Disposable Records
Supply Side Questions n Types of Data? n Who Collects Data and Why? n How Complete Are Data? n Where Do Data Go? n Walking the Process
Supply Side Questions to Ask n What Kinds of Records or Data Do You Keep on First Year Students? n How Complete Are Data Collected? n How Are Data Entered? n What Schedules Govern Your Data Collection? n How Are Data Updated?
Finding Existing Data n Follow Student Footprints >Time-Based Investigation n Who Collects Data? >Structures-Based Investigation n Why Are Data Collected? >Functions-Based Investigation
Supply Side: Items to Collect n Actual Forms and Questionnaires n Data Element Dictionaries n Data Element Definitions n Database Structures and File Formats
Demand Side Questions n Who Are the Key Constituencies? n What Are the Existing Reports and Requirements? n To Whom Are Data and Information Reported? n What, and When, Are the Decision Cycles? n How Current and Accurate Does the Information Need to Be? n What Are the Gaps in Existing Data?
Demand Side Questions to Ask n To Whom Do You Report Data and Information? n What Information Do You Need or Wish You Had? n What Are the Gaps in Data? n How Are Data Used By Others?
Demand Side: Items to Collect n Representative Reports to Constituents n Copies of External Data Reporting Requirements
Looking for... n Completeness of Data Gathered n Availability of Data n Integrity of Data n Consistency of Data Definitions n Who Coordinates Data Collection? n Who Coordinates Use of Data? n Who Controls Data and Data Processes?
Data Audit Output n Synthesized and Coherent Picture of Existing Data >Data Element Lists >Data Definitions >Data Locations and Locus of Responsibility >Data Collection Timetables >Plan for Common Data Collection/Sharing >Identification of Unmet Data Needs
Design of Recommended Data Structure n Define Core Data Elements n Responsibilities and Data Flows n Core Indicators and Calculational Routines
Data Leverage Points n Problem Identification >Persistence Rate >Violations of Prerequisite Sequencing >Student/Advisor Ratios n Context Setting >Achievement by Gender or Race >Patterns of Student Performance >Entering Student Characteristics n To Inform Discussion >Overall Patterns Rather than Anecdotal or Single n Selling Decisions >Creating Buy-In for Action
Using Evidence to Stimulate and Manage Change n Start with Obvious Discrepancies >Between: Perceptions and Reality Designs and Delivery Expectations and Results >Among Different Constituencies and Stakeholders n Avoid Perfect Data Fallacy >Recognize There Will Always Be Errors >Use Triangulation and Multiple Indicators >Know How Good Is Good Enough n Use Information in Layers >Avoid Excessive Complexity >Package Data Around Problems >Disaggregate as Needed in Response to Questions and Opportunities n Use Information to Start Discussions, not Give Answers >Develop Multiple Interpretations >Establish Data Dialogue
The Data Audit Toolkit (coming fall 2002) n Administrative Rationale >A non-technical overview and rationale for a data audit. n Technical Manual >A detailed guide, listing data elements and data, as well as step-by-step plans for a data audit.
Thank You! n Karen Paulson, 303.497.0354, Karen@nchems.org n Mike Siegel, 828.877.6009, firstname.lastname@example.org If you have questions or want to discuss the data audit and analysis project further, please contact: