ABET Assessment One Department’s Experience Paul H. Schimpf penguin.ewu.edu/~pschimpf Eastern Washington University Assessment Colloquium April 26, 2013

Caveats  This represents just one person’s experiences and opinions –I am not an assessment specialist, nor do I wish to be –feel free to disagree –debate assessment strategy and methods in your units  We are all assessment experts –most of us have probably spent more time thinking formally about student assessment, as opposed to course or program assessment –but most of us DO perform course and program assessment informally all the time  There are many ways to successfully go about assessment

Some Guiding Principles  Make it sensical and meaningful –I personally would rather see a refusal than a fake  Make it flexible –a valid outcome of assessment is making changes to assessment (it’s a meta process and can be applied to itself)  Minimize the workload –try to use data that you are already collecting –data analysis will add some work, so keep it flexible –don’t forget that we have that little item called “Service”  Empower the faculty –they are likely assessing their coursework already –don’t force them to change what they do, just ask them to document it with as little impact on them as possible

What is Assessment?  Tests and Grades constitute assessment, right? –yes, but they focus on assessment of students –do not provide the data that we are looking for in this context  Surely all faculty members, while grading an exam, have encountered this situation: –“Hmmm, very few have answered this particular question correctly. Even many that I think are my better students got it wrong, but this is an important concept.” –“Perhaps I didn’t cover this clearly? Maybe I should try covering this again, with a quick review of a pre-req, or perhaps from a different angle, or with different examples that start with something they may be more familiar with.”  Or maybe this situation: –“Everyone got this correct. Maybe that means I can trim out a little time spent on it in order to cover this other concept I’ve been wanting to add to this class.”

What is Assessment?  Whenever a faculty member is refining the content or delivery of a course, assessment is happening –It is about whether the conduct of the course could or should change. This creates one possible avenue for continual improvement. –It is about specific objective(s) of the course. Grade statistics for the entire course do not provide this information, nor do grade statistics for an entire exam (unless perhaps the exam covers only a single objective of the course). –It is about whether students are adequately mastering or understanding some objective. An average numerical score across students may obscure that a few did very well, and a whole lot more performed inadequately. –It is more powerful to know what percentage of students scored “adequately” than to know the average score.

What do you expect ME to do?  “Most of my students failed this. What am I supposed to do about it? I just got a bunch of crappy students.” –That is one possibility, but is it the only possibility? –Is this a one-time occurrence or a recurring theme? –Is this with regards to something extremely fundamental to the course, or something esoteric? –Is this a basic concept, or an advanced concept? –Are you assuming any pre-requisite knowledge that isn’t explicit in the course pre-reqs? –Is it possible that these students came out of a pre-req without the knowledge you assumed? –Have you tried taking another approach?

Waste of Time?  “I don’t believe anyone reads this and I don’t get any feedback” –“I thus conclude that it is a waste of time”  First and foremost, the reports DO need to be read by someone in your unit –if the department is not interested in continual improvement against its own objectives, then it is indeed meaningless  Don’t expect detailed feedback from an outside assessment enforcer –unless you aren’t doing any at all –or are not demonstrating that you are using assessment as part of a process for continual improvement –or are making claims in your assessment report that look suspicious

Who Closes the Loop?  Be careful what you wish for!  Where do the domain experts reside?  Who are the most important evaluators of assessment reports?  Who should recommend changes to programs, course structure, or assessment practices, based on assessment data?  Do you want to put that into the hands of a non- expert in your domain?  Close the loop yourselves, and document it so others can see that you are doing so.

How to Get Started  I describe a “top down” approach –but a bottom up analysis of program outcomes that existing course objectives lead to might also be useful  Start by defining some meaningful outcomes for your program –things that you want your students to understand, or be able to do, or be prepared for, by the time they complete –you will need to think about measurability, but at the program level you need not worry too much about that  Exam the various events that students go through to identify possible points of assessment –many assessment opportunities will naturally fall into the classroom, but those are not your only opportunities

Some Assessment Opportunities  Professional or Other Terminal Exams  Advancement Exams  Senior Capstone Presentations –you can have observers rate various aspects of performance using a rubric  Internship Reports  … ?  and yes, coursework  Your objective is to assess each program outcome in at least one place –two places would be better …

A Suggestion for Courses  With the help of the course instructor(s) –create some specific learning objectives that are appropriate for the course, and … –that support (are demonstrations of) some program outcome, –and are stated as something measurable  IMPORTANT –make sure it is understood that these are not necessarily the only objectives of the course –an instructor can add other objectives, and can choose to assess them for personal use or not  Example learning objectives –students will know how to titrate an appropriate amount of ethanol, … –students will know how to program responses to various GUI events, such as button presses, menu events, …

Empower the Instructor  Create a simple AND flexible report form  State the agreed-upon learning objective AND the program outcome that it supports  Allow the instructor to explain how they assess that and what the results are in whatever form they wish  Allow the instructor to explain any conclusions they drew from their assessments and any actions they took or suggest to the department

CS Course Assessment Form …

IMPORTANT  It is NOT a bad thing to report…  That an objective was not met –hopefully some thoughts and suggestions on the matter are offered  That an objective was not measured because the instructor feels it is an inappropriate objective for the course –hopefully this will lead to a discussion amongst the course instructors as to appropriate objectives  That all objectives were met –this is the end goal after all

Other Suggestions  Create an Assessment Plan, and Follow it –It should list your program outcomes –It should explain where your assessment points are, which outcomes they assess, and what manner of assessment is used –It should explain the process you follow, including: how often you perform each assessment how you process the data what documents are produced and how they are processed how long it takes to cycle through assessment of all of your program outcomes –and perhaps more the CS plan also covers “Educational Goals” and how they tie to “Program Outcomes” the CS plan also explains how our Program Outcomes relate to ABET “attributes” of program outcomes

CS Assessment Process