1. A Process for the Direct Assessment of Program Learning Outcomes Based on the Principles and Practices of Software Engineering Robert W. Lingard California State University, Northridge ASEE Annual Conference June 25, 2007

2. Outline  The Importance of Assessment  Problems with Assessment  Goals of an Assessment Process  An Approach to Assessment  Applying the Software Engineering Paradigm  The Annual Assessment Process at CSUN  Conclusions

3. Importance of Assessment  Assessment is required by accrediting boards (e.g., ABET).  Those with vested interests (financial and otherwise) in education are demanding accountability.  Without assessment it is difficult to know what changes to make to improve learning.

4. Problems with Assessment  Pressure for assessment can cause a “rush to assessment,” and produce meaningless results.  Pressure to “close the loop” can cause premature decisions regarding program changes.  There is a tendency to only assess high performance areas in order to “showcase” a program.  Results are often not validated.  The effects of changes are often not assessed.

5. Goals of an Assessment Process  To facilitate the direct assessment of student learning  To measure student retention and the ability to apply what they have learned  To Provide mechanisms to ensure the continuity of the process  To be an efficient and natural extension of normal operations  To satisfy the requirements of ABET and the University for assessment

6. Approach to Assessment  “The first and only goal [in education is to] teach for long-term retention and transfer.” – Diane Halpern  The best assessments of student learning are based on direct measures of achievement.  How well students have learned is best assessed by looking at their ability to apply what they learned earlier to new situations.

7. Requirements for a Complete Assessment Process  It must be comprehensive – must cover the full range of learning outcomes  It must include multiple judgments – multiple sources of evidence must be used  It must include multiple dimensions – different facets of student performance must be included  It must collect direct evidence – direct measures of student attainment must be used

8. The Software Engineering Paradigm  Requirements Analysis (understanding the problem)  Software Design (planning a solution)  Implementation/Coding (carrying out the plan)  Testing/Validation (making sure the solution is correct)

9. Understanding the Problem  Before beginning assessment, the first step is to decide what to assess – i.e., pick the most important things to assess.  Collecting information from faculty, students, alumni, and employers can give hints as to where learning problems exist.  Surveys, faculty meetings, and student interviews are ways of collecting this information.

10. Planning a Solution  Once a learning outcome is selected, a plan for conducting the assessment must be developed.  It must be determined how the assessment will be done – i.e., embedded questions in exams, a standard instrument, etc.  How the results will be evaluated must also be determined – e.g., rubrics must be developed.  It must also be determine who will do the various tasks required and when they will be done.

11. Carrying Out the Plan  The steps of the plan must be carried out, and the plan must be monitored to ensure successful completion of the assessment.  Someone must be designated as the lead, and this person has the responsibility for monitoring the plan.

12. Making Sure the Solution is Correct  Assessment results obtained must be validated – this step is often omitted.  One way to validate results is to make comparisons among several independent assessments using different approaches.  Failure to validate the results can result in program changes that may not be appropriate.  After program changes are made the effects of the changes must be assessed.

13. Iteration  Like software engineering this process is iterative.  Once program changes have been made, they must be assessed to determine whether the desired result has been accomplished.  Analysis of this reassessment might indicated the need for further changes.

14. The Annual Assessment Process at CSUN 1. Department forms Assessment Committee The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee. 2. Assessment Committee recommends outcomes to be assessed The Assessment considers results from previous assessments, informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed. 3. Department approves assessment goals and assessment plans are prepared After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities.

15. The Annual Assessment Process at CSUN 4. Department approves assessment plans and assessments are conducted Program Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared 5. Assessment are analyzed and program changes are recommended The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes 6. Department reviews recommendations and makes program changes that are determined appropriate At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole.

16. Conclusions  The process has been used successfully for program assessment at CSUN.  It has been embedded into the normal operations of the department and become accepted by faculty.  Since all faculty are involved, no one has become overburdened.  It has provided an effective means for directly assessing student learning with a focus on retention and transfer.