Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy1 CONFERENCE University towards E-learning: a Focus on Finland, France and Italy Rome May 31st, 2006 An Italian Proposal to Deal with Quality in E-learning Alfredo Squarzoni Università di Genova CRUI Steering Commitee on Evaluation

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy2 Presentation  Introduction on CRUI activity in the field of programme assessment and certification  CRUI Model for quality assessment of e-learning programmes  CRUI Requirements for certification  CRUI Procedure of assessment and certification

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy3 Introduction on CRUI Activity in the Field of Programme Assessment and Certification

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy4 Assessment Procedure finalised to first of all promote the quality improvement of a certain product, process or service Procedure finalised to first of all promote the quality improvement of a certain product, process or service

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy5 Programme Quality Level of achievement of objectives set out consistent with the needs of all interested parties in the programme Level of achievement of objectives set out consistent with the needs of all interested parties in the programme

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy6 CAMPUS-CRUI Project - between 1995 and Diploma programmes involved, mainly in engineering

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy7 CAMPUS One Project CAMPUS One Project - academic years 2001/02 to 2003/04 - over 500 programmes involved of all cultural areas

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy8 Certification Statement by an independent third party who declares, with reasonable reliability, that a certain product, process or service is in accordance with a specific rule or a specific model Statement by an independent third party who declares, with reasonable reliability, that a certain product, process or service is in accordance with a specific rule or a specific model

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy9 Accreditation Statement by a recognised institution who declares, formally and publicly, the capability of an organism to carry out specific functions or, in other words, to satisfy pre-defined standards Statement by a recognised institution who declares, formally and publicly, the capability of an organism to carry out specific functions or, in other words, to satisfy pre-defined standards

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy10 CRUI “Certification of Quality“ Not only “conformity” to the CRUI Model, but also: educational and learning objectives of value, educational and learning objectives of value, an educational process consistent with learning objectives, an educational process consistent with learning objectives, resource availability adequate to the achievement of such objectives, resource availability adequate to the achievement of such objectives, results consistent with educational and learning objectives, results consistent with educational and learning objectives, a management system capable of promoting continual improvement of the programme a management system capable of promoting continual improvement of the programme

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy11 CRUI Model for quality assessment of e-learning programmes

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy12 CRUI Model (1) It is the result of the experiences of assessment made in our country so far and of the continual attention to the assessment models of the other European countries It is the result of the experiences of assessment made in our country so far and of the continual attention to the assessment models of the other European countries

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy13 CRUI Model (2) The Model has been defined for each of the following types of programmes: “corsi di laurea” (bachelor level) and “corsi di laurea” (bachelor level) and “corsi di laurea magistrale” (master level); “corsi di laurea magistrale” (master level); postgraduate programmes; postgraduate programmes; delivered in presence, on-line or in blended form

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy14 Aim of the CRUI assessment of programme quality To promote the continual improvement of programme quality by pointing out strong and weak points

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy15 Because Quality regards  the “product”  the “system” the CRUI Model for the assessment of programmes sets out to assess both the the CRUI Model for the assessment of programmes sets out to assess both the  educational service (product assessment) and the and the  management system thereof (system assessment)

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy16 With reference to the With reference to the educational service, educational service, assessment is concerned with the degree at which all the students are provided with adequate opportunities to achieve assessment is concerned with the degree at which all the students are provided with adequate opportunities to achieve a level of learning consistent a level of learning consistent with the objectives set in the time set with the objectives set in the time set

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy17 With reference to the management system, assessment is concerned with the degree at which the system permits the constant achievement of the objectives set in the time set, i.e. the degree at which the system makes systematic the product quality

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy18 Fundamental Objectives of the assessment (1) The overall assessment of the programme quality has to check whether: programme educational objectives are consistent with the mission of the university and the needs of all interested parties (such as students, teachers, businesses and so on) and programme learning outcomes are consistent with the programme educational objectives; programme educational objectives are consistent with the mission of the university and the needs of all interested parties (such as students, teachers, businesses and so on) and programme learning outcomes are consistent with the programme educational objectives;

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy19 Fundamental Objectives of the assessment (2) academic and support staff, facilities, financial resources and cooperation agreements with businesses, research institutions and other Higher Education Institutions are adequate to accomplish programme outcomes; academic and support staff, facilities, financial resources and cooperation agreements with businesses, research institutions and other Higher Education Institutions are adequate to accomplish programme outcomes; curriculum and related processes ensure achievement of programme outcomes; curriculum and related processes ensure achievement of programme outcomes;

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy20 Fundamental Objectives of the assessment (3) appropriate forms of assessment attest the achievement of programme outcomes; appropriate forms of assessment attest the achievement of programme outcomes; the management system is able to assure the systematic achievement of programme outcomes and the continual improvement of the programme. the management system is able to assure the systematic achievement of programme outcomes and the continual improvement of the programme.

Dimensions of Assessment Consequently, the Model identifies five Dimensions of assessment : Management System and Management System and Organisational Structure; Organisational Structure; Needs and Objectives; Needs and Objectives; Resources; Resources; Educational Process; Educational Process; Results, Analysis and Improvement. Results, Analysis and Improvement.

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy22 Dimension (5) Structure of the CRUI Model QUESTIONS E L E M E N T S (18) FACTORSFACTORS FACTORSFACTORS FACTORSFACTORS QUESTIONS FACTORSFACTORS

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy23 Dimension 2: Needs and Objectives Element 1: Needs of Interested Parties QuestionsFactors Have the needs of the interested parties been identified? Interested parties identified Interested parties identified How contacts with the interested How contacts with the interested parties are maintained parties are maintained Identified needs of the interested Identified needs of the interested parties parties

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy24 Needs and Objectives Satisfaction STUDENT GRADUATE Management System and Organisational Structure Resources Educational Process Results, Analysis and Improvement Stakeholders Needs Stakeholders

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy25 Process Approach CRUI Model adopts the process approach promoted by ISO 9001: 2000 Norm Management Systems for Quality. Requirements

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy26 PDCA Method To manage a specific process, the CRUI Model suggests the PDCA method : Plan: to establish the objectives and the processes necessary to obtain results in accord with the objectives; Plan: to establish the objectives and the processes necessary to obtain results in accord with the objectives; Do: to implement the processes; Do: to implement the processes; Check: to monitor and measure the processes and the results in comparison with the objectives; Check: to monitor and measure the processes and the results in comparison with the objectives; Act: to adopt actions for the continual improvement of the process performance. Act: to adopt actions for the continual improvement of the process performance.

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy27 Dimension “Management System and Organisational Structure” Elements Management System Management System Organisational Structure Organisational Structure Re-examination Re-examination

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy28 Dimension “Needs and Objectives” Elements Needs of Interested Parties Needs of Interested Parties Educational Objectives Educational Objectives Programme Outcomes Programme Outcomes Policies Policies

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy29 Dimension “Resources” Elements Academic and Support Staff Academic and Support Staff Facilities Facilities Financial Resources Financial Resources External Relationships External Relationships

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy30 “Objects” to be assessed specific for on-line programmes Teaching Staff Teaching Staff - Adequacy of teaching staff to the needs of on-line teaching - Ability to promote on-line learning and communication with and among students - Presence of educational initiatives to favour development of on-line teaching and learning competencies

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy31 Teaching Support Staff and Technical Staff Teaching Support Staff and Technical Staff - Availabily of “content tutors”, i.e. personnel dedicated to on-line didactics, expert in programme content and in the management of on-line communication and of virtual groups, to follow students in their study activity and able to promote learning - Availability of “instructional designers”, i.e. designers for the e-learning environment, able to co-operate with and guide teachers in the definition of educational patterns to be delivered on-line

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy32 - Availability of “web editors”, i.e. experts in the production of didactic material for the digital environment; - Availabilty of “web managers”, i.e. technicians able to assure an adequate technical assistance to teachers and students - Availability of personnel for the “technical help”, i.e. technicians able to garantee the operation of the technological platform - Presence of educational initiatives to favour development of skills and competencies of teaching support staff and of technical staff

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy33 Facilities Facilities - Availability of a technological platform easily accessible, easy to use, reliable, flexible and so on, integrated in the technological infrastructures of the Atheneum - Availability of virtual rooms and virtual technological resources (laboratories, libraries and so on) adequate to the development of teachers’ teaching activity and students’ learning activity - Capacity of the virtual facilities to adapt themself, in quality and quantity, to the number of users and to the different activities developed by the programme

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy34 Dimension “Educational Process” Elements Planning Planning Student Access Student Access Delivery Delivery Support Services Support Services

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy35 Factors of particular importance for on-line programmes - Didactic coordination among teachers and teachers and teaching support staff - Adequacy of the delivery planning to favour students’ learning - Adequacy of the methodologies of delivering and assessment of students’ learning - Presence of adequate orienteering and tutoring

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy36 Dimension “Results, Analysis and Improvement” Elements Results Results Analysis Analysis Improvement Improvement

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy37 Factors of particular importance for on-line programmes - Presence of adequate criteria for the assessment of the programme results (which must take in proper account the tipology of students enrolled)

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy38 Requirements for for certification certification

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy39 CRUI certification has the main aim to attest, formally and publicly, the adoption by a programme of the CRUI Model as model for the management of the programme itself and that the programme: has educational and learning objectives of value, i.e. coherent among them, with possible requirements and with the needs of the interested parties;has educational and learning objectives of value, i.e. coherent among them, with possible requirements and with the needs of the interested parties; has at disposal resources adequate to the achievement of the educational and learning objectives;has at disposal resources adequate to the achievement of the educational and learning objectives;

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy40 has an educational process and support services which effectively allow the achievement of the educational and learning objectives; has an educational process and support services which effectively allow the achievement of the educational and learning objectives; its results attest the achievement of the educational and learning objectives set; its results attest the achievement of the educational and learning objectives set; has a management system and an organisational structure which favour the achievement of the programme objectives and promote the continual improvement of the effectiveness of the processes for the management of the programme itself. has a management system and an organisational structure which favour the achievement of the programme objectives and promote the continual improvement of the effectiveness of the processes for the management of the programme itself.

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy41 Dimension Management System and Organisational Structure Engagement in favour of a management for quality of the programme Identification of the processes for the programme management and of their interactions Management of programme documentation Management of communication with the interested parties Identification of responsibilities for the management of the programme Re-examination of the management system and of the organisational structure

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy42 Dimension “Needs and Objectives” Identification of the needs of interested parties Coherence of the programme educational objectives with the needs of interested parties Coherence of the programme learning outcomes with the programme educational objectives Policies for students

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy43 Dimension “Resources” Adequacy of teaching staff Adequacy of teaching support staff and technical and administrative staff Adequacy of facilities Adequacy of relationships with outside organisations (only for programme with vocational objectives) Adequacy of partnerships with Athenaeums of other countries (only for programmes with specific internationalisation policies)

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy44 Dimension “Educational Process” Coherence of curriculum with programme learning outcomes Adequacy of delivery planning Adequacy of prerequisites required for access to the programme Adequacy of criteria used to manage students’ careers Control of the delivery process Organisation and management of support services

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy45 Dimension “Results, Analysis and Improvement” Results relative to entrance students Results of the educational process Results relative to insertion into the job market of those students who have obtained an academic qualification (only for programmes with vocational objectives) Analysis of programme results Continual improvement of the effectiveness of the processes for the programme management

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy46 CRUI Procedure of Assessment and Certification

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy47 Assessment Methodology Self-Assessment followed by followed by External Assessment External Assessment

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy48 The procedure of assessment and certification is articulated in the following main phases: self-assessment, which involves the compilation of a self-assessment report by the programme; self-assessment, which involves the compilation of a self-assessment report by the programme; external assessment, run by an assessment team composed of two people at least, one academic at least and one representative from non-academic world at least, which winds up with the compilation of an assessment and certification report; external assessment, run by an assessment team composed of two people at least, one academic at least and one representative from non-academic world at least, which winds up with the compilation of an assessment and certification report; decision of certification, proposed by a Steering Committee of experts and assumed by a joint Board of people from academic world and people from non academic world. decision of certification, proposed by a Steering Committee of experts and assumed by a joint Board of people from academic world and people from non academic world.

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy49 To record judgement of the achievement of individual requirements, a scale with the following four grades is used: 4, in case of requirement fully fulfilled; 4, in case of requirement fully fulfilled; 3, in case of requirement fulfilled “more yes than no”; 3, in case of requirement fulfilled “more yes than no”; 2, in case of requirement fulfillled “more no than yes”; 2, in case of requirement fulfillled “more no than yes”; 1, in case of requirement not fulfilled. 1, in case of requirement not fulfilled.

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy50 Certification of conformity to the CRUI Model is awarded only if all the requirements are fully or “more yes than no” fulfilled. Where requirements are not fully met, certification is subordinated to the engagement on the part of the programme to satisfy all the remarks evidenced in the certification report by the assessment team, with the indication of the ways and of the time (no longer than 24 months) taken for their solution. The certification of conformity is valid for six years. After three years the certified programmes are subjected to a surveillance visit.

Rome May 31st, 2006 University towards E-learning: a Focus on Finland, France and Italy51 The CRUI System for assessment and certification of university programmes: a European system The CRUI System for assessment and certification of programme quality has been compared with the requirements of the document Standards and Guidelines for Quality Assurance in the European Higher Education Area, proposed by ENQA in February 2005 and adopted by the Conference of European Ministers of Higher Education in the Bergen meeting (May 2005). The comparison has shown that the CRUI System fulfils all the requirements of the ENQA document. European system It is therefore possible to state that the CRUI System can be considered a “European system” for the assessment and certification of university programmes.