1. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators Kerstin V. Siakas Alexander Technological Educational Institute of Thessaloniki, Department of Informatics, Greece siaka@it.teithe.gr

2. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 2 Structure of presentation Towards an Open European Higher Education Area Meeting the challenges – a pilot study of the department of Informatics, ATEI of Thessaloniki, Greece Experiences from the design of the 1.Self-Assessment System Design of  the self-assessment process  Key Performance Indicators (KPIs) 2. Quality Assurance System Investigating  ISO 9001:2000  Balanced ScoreCard (BSC) Lessons learnt Future work

3. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 3 Towards an Open European Higher Education Area The Bologna Process oA common frame for diverse national systems oTransparent and mutually recognised educational systems oA National Quality Assurance Systems required by 2005 in the member states including: A definition of the responsibilities of the bodies and institutions involved Evaluation of programmes or institutions, including internal assessment, external review, participation of students and the publication of results A system of accreditation, certification or comparable procedures, international participation, co-operation and networking

4. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 4 Aims and objectives of a quality assurance system in HE to create a ground for ovisibility into the processes that support the study programme omeasurements of learning outcome, capabilities and competences (what the graduate is able to do) to support a system of continuous improvement Consistent with the principle of institutional autonomy, the primary responsibility for quality assurance in HE lies with each institution itself

5. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 5 Quality Assurance The educational assessment is established around four questions : oWhich are the objectives of the educational institution? oHow does it try to achieve them? oHow does it know that it has succeeded in the achievement of the objectives? oWhat are the changes for success?

6. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 6 The department of Informatics of the ATEI-Thessaloniki, Greece Meeting the Challenge of the BOLOGNA PROCESS A pilot project sponsored by the European Programme EPEAEK II (Operational Programme "Education and Primary Vocational Training") programme was carried out 2005-2006 The working group included lecturers & students Aims of the project was the creation of a system for: oCurriculum Assessment oQuality Assurance Final report submitted July 2006

7. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 7 Phase One: Preparation Project Planning Literature Review Key Indicator Establishment Goal Question Metric Methodology Definitions of goals milestones roles and responsibilities The Bologna Process Assessment in HE Existing Laws & Regulations Educational Quality Systems

8. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 8 Activities Decision about goals and objectives of the assessment (why to measure) Team building, roles and responsibilities (who will do what - assignments) Setting up time-table and milestones (integration of tasks) Ensuring that educational processes and procedures are understood (agreement on vocabulary and meaning of different educational issues) Selection of research instrument (how to collect data) Selection of research population (where to find information, whom to ask) Design of Key Performance Indicators (KPIs) (what to measure) Design of targets for comparison (what is the meaning of measures)

9. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 9 Design of Key Performance Indicators (KPIs): Using the Goal-Question-Metric (GQM) Methodology Goal: Continuous briefing on new products / technologies Question: How many presentations / informative seminars are realised? Metrics: Seminars organised by companies, department, academic staff, students Question: What is the participation rate in research? Metrics: Participation of academic staff (and students) in research programmes Number of publications

10. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 10 KPIs: Study Programme (Curriculum) Theory – tutorials – labs oKnowledge Coverage of the discipline of Informatics oWorkload oQuality of study material oAssessment methodology Industrial placement & Final year project oSuitability to study programme oWorkload oScientific Level oAssessment methodology Accomplishment of market requirements Accomplishment of requirements for postgraduate studies

11. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 11 KPIs: Academic Staff Availability of academic staff Level of Knowledge in the discipline of Informatics Teaching ability Motivation ability Research potential Use of Technical support material Building Technological KPIs: Infrastructure

12. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 12 KPIs: Student Support Secretarial Information Organisation of seminars, workshops etc. Socrates – Leonardo da Vinci Careers office Library

13. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 13 Phase Two: Data Collection & Analysis Data Collection Data Analysis Questionnaires Interviews Observation Databases Quantitative, SPSS Qualitative

14. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 14 The Survey QuestionnairesInterviews Undergraduates205 Erasmus/Socrates Students 7 Academic Staff67 Graduates51 Employers33 Secretarial and Technical Staff 6

15. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 15 Phase Three: Continuous Improvement Dissemination of Results Defining of Action Plan Cater for commitment financial support what to improve Reports Internal meetings Workshops Publications Conferences, Journals

16. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 16 Experiences from the self-assessment Advantages oImproved experiences in assessment oImproved understanding of indicators for the quality assurance system oEasier appreciation of necessary changes Disadvantages oIncreased workload oLack of commitment from management and colleagues

17. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 17 Design of the Quality Assurance System (QA) The results from the self-assessment was the underlying base for design of the QA system SWOT analysis to find the Strengths - Weaknesses- Opportunities–Threats of existing processes and practices The objectives and the requirements of the system have to be clear before the design of the Quality Assurance System A broader societal view is required The question is not if students are ready for the educational institutions and processes but whether the institutions and the processes are ready for the students

18. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 18 ISO9001:2000: Preparation and design Preparation and design Understanding of the requirements Assessment of the situation Proposals Assurance of commitment Action plan Plan Do Check Act Deming’s P-D-C-A Cycle to follow up that: the action plan is accomplished the educational institution has succeeded in the achievement of the objectives

19. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 19 ISO 9001:2000 Application Policy and management assurance Selection and training of management representative and application leader Internal review Improvements of documentations Selection of accreditation body Preliminary visit Preliminary assessment and preparation stage Assessment Accreditation Maintenance

20. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 20 The Balanced Scorecard (BSC) Methodology Four Perspectives: Financial Customer (students) Internal procedures (the internal procedures of the institution) Learning and improvement (ability of constant training aiming at continuous improvement and competitiveness) Steps: The institution sets goals & targets for each of the perspectives collects the evidences to verify the performance and quality level (questionnaires, databases etc.)

21. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 21 Lessons learnt from the pilot study The transformation from bad practices to survival and competitive success require: oAn institutional culture change oManagement commitment

22. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 22 The situation today in the institution Assessment is a fact A QA committee on institutional level is created Nation wide questionnaires were distributed in Sept 2007 to all HE institutions in Greece to be used as self-assessment Every department is asked to create a QA group to take action (within their normal duties) on the new directives

23. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators 23 Further work In the department of Informatics oThe creation of the QA group oDiscussions of the results from the previous self- assessment oInvestigation of the coverage of key performance indicators of the new nation wide questionnaire oDecide on questionnaire to use (in its current form or with extension of other questions) oCreation of an on-line version of the questionnaire oCarry out assessment within the academic year 2007- 2008 oDissemination of results

24. _________________________________________________________________ GIGP2007, SPLIT, 17-19 Oct 2007, The Design of a Quality Assurance System in Higher Education – Selecting Key Performance Indicators Thank You! Questions? Kerstin V. Siakas siaka@it.teithe.gr