1. The Development of a Comprehensive Assessment Plan: One Campus’ Experience Bruce White ISECON 2007

2. Feedback / accountability 1. “For society to work […] we must be accountable for what we do and what we say.” 2. “No person can succeed unless he or she is held accountable” 3. “Feedback is the breakfast of champions” 4. “You need a culture of assessment, not a climate” 1 -Betty Dowdell 2 – Grant Wiggins 3 – Ken Blanchard 4 – Gloria Rogers

3. Overview Are we teaching what we say we are? Are students learning? How can we be more effective in our instruction? SO … What do we want students to learn? Why do we want them to learn it? How can we help them to learn it? How do we know what they have learned?

4. Furthermore...  Stakeholders want to see if we are accomplishing our goals of education.  Possible stakeholders:  Students  Parents  Employers  Board of Regents / State Agencies  Accrediting groups (AACSB / ABET / etc.)  Faculty  Alumni

5. Our campus program  The Information Systems Management program at Quinnipiac University in Hamden Connecticut started our journey towards a comprehensive assessment program in 2003.  Prior ‘assessment’ was informal:  So ISM faculty – how do you think we are doing?  So Advisory Board – what advise do you have for us?  So IS education community – what should we teach (like IS2002 curriculum)  So Employers – what do our students need to know (or know better)  Etc.

6. Our desired outcomes:  Analysis and design of information systems which meet enterprise needs.  Use and experience with multiple design methodologies.  Experience in the use of multiple programming languages.  Development of hardware, software and networking skills.  Understanding of data management.  Understand the role of IS in Organizations.

7. Possible assessment methods: Direct Assessment Methods :  Simulations  Behavioral Observations  Performance Appraisals  Locally Developed Exams  External Examiner  Portfolios / E-portfolios  Oral exams  Standardized Exams (Source: Gloria Rogers- ABET Community Matters 8-06)

8. Indirect Assessment Methods  Exit and other interviews  Archival data  Focus groups  Written or electronic surveys / questionnaires  Senior exit surveys  Alumni surveys  Employer surveys  Other factors:  IS model curriculum  Advisory board  Alumni Surveys

9. Foundation of Our Assessment Program We got interested in the CCER IS Assessment test early It is a direct assessment test based on the IS2002 model curriculum It has been thoroughly tested and analyzed It has been shown to be valid and reliable Test scores reported on 37 different areas – and relevant to our learning outcomes The test questions are written at higher levels of Bloom’s taxonomy – with scenarios

10. More on our assessment process  We also use a senior exit survey (indirect measure)  An advisory board gives input  Informal controls:  Campus decisions (such as number of credits allowed, changes in general education)  Model Curriculum Changes  Employer input  Conferences / technologies

11. Specific Learning Skills Skill Set 3.0 Strategic Org. Systems Develop. 2004200520062007Avg 3.1 Organizational Systems Development 3.1.1 Strategic Utilization of Information Technology 3740463940.5 3.1.2 IS Planning3731471432.3 3.1.3 IT and Org. Systems3433502936.5 3.1.4 Information Systems Analysis & Design 4742534446.5 3.1.5 Decision Making2322 1721 3.1.6 Systems Concepts, Use of IT, Customer Service 4335453840.3 3.1.7 Systems Theory and Quality Concepts 4338432036

12. 3.2 Project Management 2004200520062007Avg. 3.2.1 Team Leading, Project Goal Setting 4942613647 3.2.2 Monitor and Direct Resources and Activities 354150 44 3.2.3 Coordinate Life Cycle Scheduling and Planning 5564744559.5 3.2.4 Apply concepts of continuous improvement 4744374242.5 3.2.5 Project Schedule and Tracking 4537574345.5 Continued

13. Overall Analysis Area 2004200520062007Avg Hardware and Software 39.638.748.031.139.35 Modern Programming Language 38.137.342.232.437.5 Data Management 41.240.953.941.444.35 Networking and Telecommunications 39.435.153.646.343.6 Analysis and Design 43.143.053.640.945.15 Role of IS in Organizations 50.144.758.745.549.75 Number taking test2724119 Overall

14. Senior Exit Survey Learning Objective 2006 scores 2007 scores Systems Analysis (including project management 4.44.5 Alternative Design Methodologies 3.73.5 Programming Languages 4.03.0 Hardware and Software 4.24.5 Networking 3.93.5 Data management 4.2 IS in Organizations 4.24.4 Ethics in IS / IT 4.23.8 Global aspects of IS / IT 3.63.8

15. Next Step – setting metrics  So … we have a solid direct measurement  And … we have a good indirect measure  Now … what  We are working on setting metrics (especially for our direct measurement – the CCER IS Assessment test)  From ABET:  “Every student does not have to achieve the desired outcomes, but targets must be defined.”

16. Setting expectations  The faculty have considered the goals of the program and feel our program emphasizes systems analysis, the role of IS in organizations and data management ISECON 2007 - Accreditation16 AreaExpectation Systems Analysis80% will score 50 or higher Programming60% will score 45 or higher Data Mgmt70% will score 50 or higher Role of IS in Org.80% will score 50 or higher Networking70% will score 45 or higher Hardware / Software70% will score 45 or higher

17. Now what  Next year as students take the CCER IS Assessment test and as we get feedback from our senior survey, we will analyze the data to see if our outcomes have been reached.  If they have:  If they haven’t:  We analyze why not – was it poor instruction? Poor students? Poor textbook? Overly optimistic expectations?  We change in an effort to ‘constantly and continually improve’ our program!!