Presentation on theme: "Establishing a Culture of Assessment Evaluation of the Effectiveness"— Presentation transcript:
1Establishing a Culture of Assessment Evaluation of the Effectiveness Alabama A&M UniversityJanuary 07, 2008John D. Jones, Ph.D.Associate Provost for Institutional Effectiveness and ResearchBennett College
2Goals for the Workshopclear understanding of how to develop an assessment planmore knowledgeable about assessment methodsmore knowledgeable about analyzing and reporting assessment resultsmore knowledgeable about how to assure the quality of assessment plans and the reporting of assessment results
3What Do We Mean by Assessment? it is a formative evaluation process designed to support improvementit is continuousit is focused on improvementstudent learningstudent developmentthe institution and its people
4Levels of Assessment classroom assessment course assessment assessment of individual students performance at the course level by an instructorscourse assessmentassessment of how well a course is meeting student learning outcomesprogram assessmentassessment of how well an academic program is meeting student learning outcomesassessment of how well a support program is meeting its objectivesinstitutional assessmentassessment of campus-wide issues or programs
5Institutional Effectiveness and Program Assessment focuses on continuous quality improvementacademic programsadministrative support unitsexamines students learning outcomes, program outcomes, customer satisfaction, and unit performance to identify areas to improveleads to actionsconducted annually
6Purposes of Program Assessment to improvethe assessment process should provide feedback to determine how the program can be improvedto informthe assessment process should inform faculty and other decision makers of the contribution and impact of the programto provethe assessment process should encapsulate and demonstrate what the program is accomplishing to students, faculty, staff, and outsidersto supportthe assessment process should provide support for campus decision-making activities such as program review, strategic planning, and external accountability activities such as accreditation
7Effective Program Assessment Should Answer these Questions What are you trying to accomplish?How well are you doing it?How, using the answers to the first two questions, can you improve what you are doing?What and how does a program contribute to the development and growth of its students and/or the support of its customers?How can student learning be improved?
8Program Assessment is Effective When Assessment… is viewed as comprehensive, systematic, and continuous activityis viewed as a means for self-improvementmeasures are meaningfulutilizes multiple measures and multiple sourcesis used as a management toolresults are valued, and are genuinely used to improve programs and processeshas sufficient coordination and reviewincludes all constituents in the process (e.g., faculty, staff, administrators, students, community, industry)
9Mechanics of Assessment assessment is a continuous improvement processto improve, you need to know where you are today and where you would like to gomission (purpose)vision (where you would like to go)goals (steps to getting where you would like to be)objectives or outcomes (what you need to achieve in order to get there)measures (how well you are currently doing)to improve, you need to take actionanalyze your program or operations to determine changesplan the changestake action
10Organizing for Assessment before assessment can begin, key players, committees and structures must be identified and assume responsibility for designing, implementing, and carrying out the assessment processunderstanding the needs of program or unit can you help think about the design of the assessment plandepending on purpose, the plan can be informal (for internal use) or formal (external audience)
11Identify the Scope of the Plan What should the assessment include?assess resources (facilities, students, faculty)assess processes (pedagogy, advising, feedback)assess results or outcomeswho/what gives you feedbackacademic program assessment typically focuses on student outcomesadministrative unit assessment typically focuses on quality of products, processes, and services
12Defining the Mission Statement a broad statement of what the program or unit is, what it does, and for whom it does ita clear description of the purpose of the program or unit and the learning environmentreflects how the program contributes to the education and careers of students graduating from the program or how the unit supports its customersaligned with department, college, and university missionsdistinctive for your program or unit
13Components of a Mission Statement primary functions or activities of the program or unitmost important functions, operations, outcomes, and/or offerings of your program or unitpurpose of the program or unitthe primary reasons why you perform your major activities or operationsstakeholdersgroups or individuals that participate in the program and those that will benefit from the program or unit
14Structure of a Mission Statement “The mission of (name of your program or unit) is to (your primary purpose) by providing (your primary functions or activities) to (your stakeholders).” (Additional clarifying statements)(Note: the order of the pieces of the mission statement may vary from the above structure.)
15Checklist for Mission Statement Is the statement brief and memorable?Is it distinctive?Does it clearly state the purpose of the program or unit?Does it indicate the primary function or activities of the program or unit?Does it indicate who the stakeholders are?Does it clearly support the department’s, college’s, and university’s missions?
16Defining Program Goals establishment of goals prior to developing objectives is recommendedsome assessment processes don’t include reporting of goalsgoals are long-term organizational or program targets or directions of developmentthey state in broad terms what the organization wants to accomplish or become over the next several yearsgoals provide the basis for decisions about the nature, scope, and relative priorities of projects and activitiesgoals should help move the organization or program to attain its vision
17General Process Used in Generating Goals examine missionthink what that unit or program would look like and how its services and operations (refer to you mission) would need to be conducted to reach that visionimprove, minimize, maximize, providegenerate list of potential goals and prioritizestate these as goalstwo general approaches to developing goalsreview existing documentsideal unit or program approach
18Defining Objectives and Student Learning Outcomes a measurable target with a time limit that must be met on the way to attaining a goalstudent learning outcomespecific statements that describe the intended learning outcomes that must be met on the way to attaining the degreetypically derived from goal statementsmore precise, more specific and measurable than the goal statementthere can be more than one objective or outcome related to each goal
19Think SMART When Defining Student Outcomes and Program Objectives Specificclear and definite terms describing the abilities, knowledge, values, and attitudesMeasurableit is feasible to get the data, data are accurate and reliable, it can be assessed in more than one wayAggressive but Attainabledon’t let the perfect divert you from what is possibleResults-oriented and Time-bounddescribe where you would like to be within a specified time perioddescribe what standards are expected from students or what percent of students are expected to achieve a particular level of performance
20Agenda Identifying Student Learning Outcomes What is a student learning outcomeWriting student learning outcomesTypes of student learning outcomesIdentifying key outcomes
21Agenda…continued Measuring Student Learning Outcomes Direct evidence Indirect evidence
22Student Learning Outcomes What is a Learning Outcome?Knowledge, skills and attitudes that students take with them from a learning experience.
23Writing Learning Goals Aim for goals that are neither too broad or too specificUse concrete action wordsDefine fuzzy termsFocus on the end, not the meansFocus on the most important goalsWork collaboratively with colleagues
24Types of Learning Goals Increase knowledge and basic understandingMemorizationReplicating a simple procedureDefining conceptsSummarizing conceptsExplaining concepts
25Types of Learning Goals Thinking, Performance, and Interpersonal SkillsAnalysis, evaluation, problem solving, and decision makingPhysical skills related to manipulating ideas, concepts, tools etc.Listening, working with people from diverse backgrounds, teamwork, leadership ability
26Types of Learning Goals Attitudes and ValuesAppreciationAwareness of one’s own values, attitudes, and opinionsIntegrityCharacterValuing learning
27Most Valued Learning goals Today! Communication skillsInformation literacy and research skillsThinking skillsInterpersonal skills
28Identifying key learning goals Identify resources for potential goalsMake it a collaborative processAchieve consensus on common goalsPreserve academic freedomShare goals with students
29Measuring Student Learning Outcomes Direct evidence of student learning is tangible, visible and self-explanatory evidence of exactly what students have or have not learned.
30Examples of Direct Evidence Ratings of student skills by their field experience supervisors.Scores and pass rates on appropriate licensure/certification exams.Capstone experiences such as research projects, theses, dissertations, exhibitions, performances scored using a rubric.Other written work or performance, scored using a rubric.Portfolios of student work.
31Examples of Direct Evidence… continued Scores on locally-designed tests such as final examinations in key courses, qualifying exams, and comprehensive exams, accompanied by test blueprints describing what the test assess.Score gains between entry and exit on published test, local test, or writing samples.Employer ratings of the skills of recent graduates.Student reflections on their values, attitudes, and beliefs.
32Measuring Student Learning Outcomes Indirect evidence of student learning provides signs that students are probably learning , but the evidence of exactly what they are learning is less clear and less convincing.
33Examples of Indirect Evidence Course GradesAssignment grades, if not accompanied by a rubric or scoring guideAdmission rates into graduate school or graduation ratesPlacement rates of graduates into appropriate careers and starting salariesAlumni perceptions of satisfaction and career responsibilitiesStudent ratings of knowledge and skillsHonors, awards, and scholarships earned by students and alumniStudent/alumni satisfaction with their learning
34More Details on Student Learning Outcomes describe specific behaviors that a student of your program should demonstrate after completing the programfocus on the intended abilities, knowledge, values, and attitudes of the student after completion of the programWhat is expected from a graduate of the program?What is expected as the student progresses through the program?
35Why Are Student Learning Outcomes So Important? basis for program improvementinstruction, course design, curricular designcommunicate instructional intentincluded in the syllabusincrease awareness of learning (for students)common languageadvising materialspromotional materialssupport accreditation and evaluation
36When Defining Student Learning Outcomes frame the learning outcome in terms of the program rather than individual courses or individual studentseach student will receive a B or better in the XYZ coursegraduates from the program will demonstrate knowledge of engineering fundamentalsuse concrete action verbs (e.g., define, classify, operate, formulate) rather than passive verbs (e.g., be exposed to) or vague verbs (e.g., understand, know)action verb lists based on Bloom’s Taxonomy
37Types and Levels of Student Learning Outcomes (Bloom’s Taxonomy) cognitive: recall and intellectual skillsknowledge, comprehension, application, analysis, synthesis, evaluationaffective: attitudes, values, interests, appreciation and feelings toward people, ideas, places, and objectsreceiving, responding, valuing, organization, characterization by, valueskills- perception, set, guided response, mechanism, complex overt response, adaptation, origination
38When Defining Student Learning Outcomes (SLO) do not to join elements in one statement that can not be assessed by single methodBSHE graduates will demonstrate knowledge of math, science, and engineering fundamentals, and gain competency in conducting oral presentations.SLO statements should focus on the learning results and not on the learning processcomputer applications will be introduced in all core engineering coursesBSHE graduates will demonstrate proficiency in XXX computer applications
39When Defining Student Learning Outcomes (SLO) SLO statements should be stated so that the outcome can be measured by more than one assessment methodstudents completing the XYZ engineering program will score over 95% on a locally-developed exam that tests application of engineering principlesstudents completing the XYZ engineering program will demonstrate competence and the ability to apply engineering principlesSLO statements should indicate the level and type of competence that is required of graduates of a program
40Checklist for Student Learning Outcomes aligned to mission and goal statementsclearly describe and define expected abilities, knowledge, values, and attitudes of the graduates of the programsimply stateddistinctive and specific to programstated so that a single method can be used to measure the outcomestated so that more than one measurement method can be usedfocus on the learning results and not the learning processmeasurable and there are resources availablecan be used to identify areas to improve
41Describing Measurement Methods What are you going to use?presentation, assignment, test, survey, observation, performance ratingOf and/or by whom?student, mentor, focus group, customer, process, courseContext (e.g., where or when)point-of-service, capstone, throughout the year, end of programFor what purposedesired objective or learning outcomeExample: Test the students at the end of the program for their level of knowledge in XYZ
42Curriculum or Course-based performance-basedcapstone coursescapstone projectscase studiesclassroom assessmentcourse-embedded assignmentscourse-embedded exam questionsportfoliosothercurriculum and syllabus analysiscontent analysis of coursesreflective essays
43After Identifying the Potential List of Measures You Need to… select the “best” onestry to identify at least two measure per objective or outcomeidentify performance targetsbalance between stretch targets versus achievable targetsexamplesurvey (using the Graduating Senior Survey) the students at the end of the program for their level of satisfaction with their communication skills (indirect method)90% or more of the students will rate their level of satisfaction with their communication skills as “very good” to “excellent” on a survey (using the Graduating Senior Survey) that the students complete at the end of the program
44Selecting the “Best” Assessment Methods relationship to assessment—provide you with the information you needreliability—yields consistent responses over timevalidity—appropriate for what you want to measuretimeliness and cost—preparation, response, and analysis time; opportunity and tangible costsmotivation—provides value to student or customer, respondents are motivated to participateotherresults easy to understand and interpretchanges in results can be attributed to changes in the program
45Challenges and Pitfalls one size does not fit all—some methods work well for one program but not othersdon’t try to do the perfect assessment all at once—take a continuous improvement approachallow for ongoing feedbackmatch the assessment method to the objective and not vice-versa
46After Identifying the Potential List of Measures You Need to… develop assessment instrumentssurveysexamsassignmentsscoring rubricsportfoliosideally you want them to be reliable, valid, and cheapapproachesuse external sourcesseek help from internal sources (e.g., Institutional Research Office)do it your selfthe instrument may need to be modified based on assessment results