1.
IMPLEMENTATION OF AN INDEPENDENT QA SYSTEM IN HE AND IT’S DEVELOPMENT: A CASE OF LITHUANIAN HIGHER EDUCATION
2017
Nora Skaburskienė
Director
Centre for Quality Assessment in HE (SKVC)

2. Higher Education in Lithuania
44 Higher Education Institutions
~ 1800 I&II cycle study programmes
2,9 mln. population
~ students
5/2/2019

3. Binary system of HE
There are 44 HEIs in Lithuania: 22 Universities 22 Colleges of higher education
college of higher education sector
Pirmą teiginį – į slaidą apie nuolatines ir ištęstines
university sector
5/2/2019

4. State vs. private HE 44 HEIs in Lithuania: non-state sector 26 state
5/2/2019

5. LTQF 8
EQF 8
LTQF 7
EQF 7
LTQF 6
EQF 6
NON-DEGREE STUDIES

6. to encourage quality of activities of HEI
Activities of SKVC
to contribute to creation of favorable conditions for free movement of persons
assessment of foreign qualifications
provision of info and consultations
to encourage quality of activities of HEI
review and accreditation of HEI (institutional review)
evaluation and accreditation of HE study programs (new and in operation)
assessment of (new) establishing HEI

7. the free movement of persons in the world with the
MISSION
To create favourable conditions for the improvement of higher education quality in Lithuania and
the free movement of persons in the world with the
aim of enhancing competitiveness of Lithuanian higher education and clarity of its qualifications
VISION
SKVC is an influential generator, implementer and disseminator of ideas on improvement of quality and enhancement of internationalisation in higher education - in both Lithuania and the rest of Europe

8. We at SKVC understand quality first of all as fitness for purpose.
Kas yra kokybė?
What is quality?
We at SKVC understand quality first of all as fitness for purpose.
Quality is multidimensional phenomenon, where agreement of all stakeholders (academics, administrators, students, labor market representatives) regarding what is quality it is of utmost importance.

9. External Quality Assurance
At least once every 6 years:
Institutional Review and accreditation
Study programme evaluation and accreditation

10. Involvement of international experts
Study programme evaluation (from 1998):
International experts - from 2002
Institutional review (from 2011):
International experts - from the beginning
5/2/2019

11. Number of evaluations by international/national experts’ groups
5/2/2019

12. Countries of experts (programme evaluation 2016)

13. Countries of experts (Institutional review 2011-2016)

14. International relations
INQAAHE (International Network for Quality Assurance Agencies in Higher Education) – full member from 1997
ENIC/NARIC (European Network of National Information Centres / Network of National Academic Recognition Information Centres) – full member from 1999
CEENQA (The Network of Central and Eastern European Quality Assurance Agencies in Higher Education) – co-founder and full member from 2000
ENQA (associate member from 2000, full member since 14 Sept 2012)
Listed on EQAR as trustworthy QA agency from 2012

15. Quality assurance system: External QA (SKVC)
- Internal QA (HEIs)
5/2/2019

16. Agreement of all stakeholders on:
Building External QA system
Agreement of all stakeholders on:
Aim and objectives of the system
Key players
Concept of quality
In shaping national quality assurance systems, many actors plaid an active role: international organizations, national authorities,
national stakeholder organizations
• QA is multi-dimensional and embedded in contexts, thus, can serve multiple purposes and multiple interest groups
• The role of the state is very visible in setting the frameworks of requirements for external quality assurance. This should not be
seen as a negative feature. It was states (or to be precise, Ministers) who subscribed for the ESG at Bergen meeting and increased
EQAR role in the most recent Bucharest meeting.
• Coordinated policy measures across EHEA undoubtedly act as powerful impetus for all of us who take international commitments
seriously.
5/2/2019

17. Key player in implementation of EQA – the Agency
Building External QA system
Key player in implementation of EQA – the Agency
Key documents for EQA – ESG and national legal acts
5/2/2019

18. ESG Part 2: Standards for external quality assurance
2.1 Consideration of internal quality assurance 2.2 Designing methodologies fit for purpose 2.3 Implementing processes 2.4 Peer-review 2.5 Criteria for outcomes 2.6 Reporting 2.7 Complaints and appeals

19. 2.1 Consideration of internal quality assurance
Quality assurance in HE is based on the institutions’ responsibility for the quality of their programmes and other provision
It is important that external quality assurance recognises and supports institutional responsibility for quality assurance.
This standard relates to the crucial link between internal and external quality assurance of higher education institutions and programmes.
To “address” means that the agency should systematically include all standards of Part 1 of the ESG in their criteria and procedures used to evaluate/accredit/audit institutions or programmes, while they may be addressed differently depending on the type of external quality assurance.
What would be important, that the context is explained and justification is given, why the agency decided to address the standards of Part I in such way (if they do it differently).
However, it is understood, that there could be various external QA procedures, some relying more on self-evaluation, others less (such as licensing procedures). As a rule, there is more variety than just institutional reviews, audits of QA systems, programme evaluations, so various other activities should be constructed taking into account the purpose for which they serve.
The QA agency methodology should clearly state that external quality assurance in major part relies on internal quality assurance of HEIs, and this in most situations means that external processes include institutional self-evaluations.

20. 2.2 Designing methodologies fit for purpose
In order to ensure effectiveness and objectivity it is vital for EQA to have clear aims agreed by stakeholders
If the agency works in different jurisdictions it should take into account the relevant regulations of the jurisdiction in which the reviewed institution is based.
In the case of joint programmes, the quality assurance agency should use the European Approach for Quality Assurance of Joint Programmes.
You have to remember that It is not a requirement to use the European Approach for Quality Assurance of Joint Programmes, it is only one of the options, and the agencies are also able to choose another methodologies.
11th ENQA TRAINING OF AGENCY REVIEWERS

21. 2.3 Implementing processes
External quality assurance processes should be reliable, useful, pre-defined, implemented consistently and published. They include:
a self-assessment or equivalent;
an external assessment normally including a site visit;
a report resulting from the external assessment;
a consistent follow-up.
The whole chain of EQA should be present.
If the site visits are not a part of the processes, there need to be clear reasons for that. If the agency in some procedures deviate from the norm, there has to be explanation and justification provided.
There could be various approaches to follow-up, it is up to the agency to determine the nature of the follow-up in light of its mission and as appropriate in its context, but the next review should not be interpreted as follow-up (ENQA expectation). It should be in between the reviews – touch base with the institution to see how is it going after the review.
Good practice – recommendations, methodologies for HEIs provided by the agency to support follow-up.
The agency remains responsible for ensuring a consistent follow-up even if the formal decision is taken by another body.

22. are carefully selected;
2.4 Peer-review experts
In order to ensure the value and consistency of the work of the experts, they:
are carefully selected;
have appropriate skills and are competent to perform their task;
are supported by appropriate training and/or briefing.
The agency ensures the independence of the experts by implementing a mechanism of no-conflict-of-interest.
Desirable - the involvement of international experts in external quality assurance
It is important how the experts ar selected, how the peer‘s groups are composed. It should be clear, what the expert selection criteria for each QA activity are.
The agency should make use of a wide range of experts with different perspectives, including those of institutions, academics, students and employers/professional practitioners. Wide range of experts – not necessarily in every panel (depending on the nature of the process), but in the pool of the agency;
It must be guaranteed that experts have not any relation with the institution assessed and are external experts.
Good practice - it could be beneficial to include experts from a variety of national origins not only in cross-border QA activities but also at home (especially in the small countries). Agencies should be encouraged when possible to include foreign experts, but this is not a requirement.
Good practice – local expert for cross-border QA reviews.
The agency should not only ensure a consistent selection procedure of experts but also appropriate training or briefing of experts.
Another important point - avoiding conflicts of interest of experts, especially when the agency operates within small scientific or professional communities.
Role of the students - agencies should demonstrate that the students are equal members of the panels. They should not necessarily have the same responsibilities as other experts, but they have to be equal members, equally participating in decision making process.

23. 2.5 Criteria for outcomes
Any outcomes or judgements made as the result of EQA should be based on explicit and published criteria that are applied consistently, irrespective of whether the process leads to a formal decision
Agencies should have tools to ensure consistency of decision making/outcomes. They have to pe prie-defined and publicly available
Important – training for experts how to apply the criterion (understanding and interpretation)
Balance – taking into account the specific context of the institution/consistency

24. Full reports by the experts should be published
2.6 Reporting
Full reports by the experts should be published
Clear and accessible to the academic community, external partners and other interested individuals
If the agency takes any formal decision based on the reports, the decision should be published together with the report
All reports should be published in full, including those that resulted in a negative decision or conclusion. Full reports should be published also for cross-border QA activities if those activities are within the scope of the ESG (even if voluntary).
The publication of summary reports (rather than full reports) does not fulfill the requirement of the standard.
All experts should be appropriately involved in producing the report.
Good practice - it is up to the agency to prepare a summary report, but this could be recommended (added value).
Good practice - it is not compulsory to give the opportunity to HEIs to point out the errors of fact in the reports, but the benefit could be underlined as it improves reliability.

25. 2.6 Reporting
Guidelines: The report by the experts is the basis for the institution’s follow-up action of the external evaluation and it provides information to society regarding the activities of an institution. In order for the report to be used as the basis for action to be taken, it needs to be clear and concise in its structure and language and to cover:
context description (to help locate the HEI in its specific context);
description of the individual procedure, including experts involved;
evidence, analysis and findings;
conclusions;
features of good practice, demonstrated by the institution;
recommendations for follow-up action.
The preparation of a summary report may be useful. The factual accuracy of a report is improved if the institution is given the opportunity to point out errors of fact before the report is finalised.
11th ENQA TRAINING OF AGENCY REVIEWERS

26. 2.7 Complaints and appeals
Complaints and appeals processes should be clearly defined as part of the design of EQA processes and communicated to the institutions
It should be possible to appeal any formal decision and agency must have a predefined process for that.
Even if the procedure is not leading to the decision, there should be a possibility to contest the outcomes or the procedures.

27. ESG Part 3: Standards and guidelines for QAAs
3.1 Activities, policy and processes for quality assurance 3.2 Official status 3.3 Independence 3.4 Thematic analysis 3.5 Resources 3.6 Internal quality assurance and professional conduct 3.7 Cyclical external review of agencies

28. 3.1 Activities, policy and processes for quality assurance
Agencies should undertake EQA activities as defined in Part 2 of the ESG on a regular basis
They should have clear and explicit goals and objectives that are part of their publicly available mission statement
These should translate into the daily work of the agency
Agencies should ensure the involvement of stakeholders in their governance and work.
Emphasis on the regularity - agencies should themselves conduct external QA activities on a regular basis.
Organisations that only occasionally organise reviews of institutions or programmes do not comply with the standard.
QA should be the main/one of the main activities of the agency.
The involvement of stakeholders should be organised in a way that ensures the agency's independence

29. 3.2 Official status
Agencies should have an established legal basis and should be formally recognised as quality assurance agencies by competent public authorities
This can be ignored if there is a requirement in the law to be first registered on EQAR in order to be formally recognised by a (national) public authority.
For international organisations it might be a prerequisite to be registered on EQAR in order to be formally recognised by a (national) public authority. In such a case, the agency is not expected to be formally recognised as a quality assurance agency before it is registered on EQAR.

30. Agencies should be independent and act autonomously.:
3.3 Independence
Agencies should be independent and act autonomously.:
Organisational independence, demonstrated by official documentation (e.g. instruments of government, legislative acts or statutes of the organisation)
Operational independence: the definition and operation of the agency’s procedures and methods as well as the nomination and appointment of external experts are undertaken independently from third parties such as HEIs, governments and other stakeholders
Independence of formal outcomes: anyone contributing to EQA activities of an agency (e.g. as expert) is informed that while they may be nominated by a third party, they are acting in a personal capacity and not representing their constituent organisations when working for the agency. Independence is important to ensure that any procedures and decisions are solely based on expertise
Where an agency's independence is not obvious from its structures and status, it should be considered in greater detail how operational independence is safeguarded in practice.
The integrity of expert groups' reports should be ensured by preventing undue influence of stakeholders on the findings, analysis, conclusions and recommendations, and that the body which takes (accreditation, audit, etc.) decisions after external QA activities, operates independently and without political or other undue influence.
If the agency has other activities than external QA (e.g. seminars, consultancy), adequate policies and processes should be in place to safeguard independence of the respective organisational units in performing their QA functions.

31. 3.4 Thematic analysis
Agencies should regularly publish reports that describe and analyse the general findings of their external quality assurance activities
It should be stressed that reports must be about general findings of EQA activities not about any other kind of information about the Agency or the performance of an agency.
Must be different from yearly activity reports.

32. 3.5 Resources
Agencies should have adequate and appropriate resources, both human and financial, to carry out their work
Agencies should demonstrate a balance between financial-human resources and the volume of their activity.
Agencies should have enough resources to carry on their mission and to constantly improve (e.g. investment in staff development).
Resources include physical conditions, material and financial resources.

33. 3.6 Internal quality assurance and professional conduct
Agencies should have in place processes for internal quality assurance
Agencies need to be accountable to their stakeholders
The review and improvement of their activities are on-going so as to ensure that their services to institutions and society are optimal
The processes for internal quality assurance need to be formal and regular, and not just informal.
There should be clear separation between ESG activities and other activities
Agency’s activities out of the scope of the ESG: to avoid confusion, agencies have to be open and honest that they are doing this other way. If they do not follow the ESG in some activities, how do they make this clear to public?

34. 3.7 Cyclical external review of agencies
Agencies should undergo an external review at least once every five years in order to demonstrate their compliance with the ESG

35. Challenges for Agencies with ESG 2015
Mandatory student involvement
Mandatory publication of full reports
International participation in panels and agency
Thematic analysis
Competition at home (for some)
Markets abroad

36. What does ESG 2015 mean for Higher Education Institutions?
Expansion of Part 1 – Standards and Guidelines for Internal Quality Assurance from 7 to 10 Standards
Policy for quality assurance
Design and approval of Programmes
Student-centred learning, teaching and assessment
Student admission, progression, recognition and certification
Teaching staff
Learning resources and student support
Information management
Public information
On-going monitoring and periodic review of programmes
Cyclical external quality assurance
Those guidelines do not provide specific problem solutions, they just indicate the
areas of interest and responsibility for universities striving towards high effectiveness and
good teaching reputation.

37. How HEIs can address internal quality assurance in line with ESG 2015
Helpful guide on how universities can address ESG 2015 (part 1)
Published (Sep 2105) by

38. You can create synergy when discussing internal quality systems.
It is impossible to copy quality as it depends on culture and academic ethics of HEI.
Chosen quality system should be aappropriate for achieving specific aims at a particular institution.

39. The key mutual element is inscribed in the Plan-Do-Check-Act Cycle
The key mutual element is inscribed in the Plan-Do-Check-Act Cycle. The internal quality
assurance system is supposed to outline the goals that the given university aspires to

40. HEIs confront a number of challenges in IQA design:
Implementation of IQA
HEIs confront a number of challenges in IQA design:
choosing an appropriate focus
integrating IQA tools into a cost-effective and coherent system
considering graduate employability
finding an appropriate balance between centralized and decentralized structures

41. Challenges (for some) with ESG 2015
1.2 – Design and approval of programmes
Programmes are designed by involving students and other stakeholders in the work
Student-centred learning, teaching and assessment
The implementation of student-centred learning and teaching:
Respects and attends to the diversity of students and their needs, enabling flexible learning paths
Flexibly uses a variety of pedagogical methods
Encourages a sense of autonomy in the learner

42. Stakeholder involvement IQA integrated with strategic planning
Success factors
Leadership support
Stakeholder involvement
IQA integrated with strategic planning
Effective management information system
Leadership support was identified by both academic and administrative staff as a necessary and commonly present factor in the case universities in facilitating the integration of centralized and decentralized management of IQA. Linking IQA with decision-making can close the loop at three levels: individual level; academic programmes, and strategic planning of the whole university. Indeed, strategic planning provides a framework of orientations and goals, including on quality, at all levels towards which IQA works most effectively if all levels are engaged.
Effectiveness of the IQA system also relied heavily on the level to which students and staff were aware of and involved in its processes and tools. For instance, programme reviews and job market analysis were found effective if they incorporated employer recommendations to revise academic programmes in line with employment needs. In terms of limits, students and staff felt that they did not receive enough feedback from certain IQA tools, such as course evaluations or student satisfaction surveys, the study found.

43. Recommendations from the experts on IQA
Part of universities does not have an integrated quality assurance system that would cover all processes within the institutions and all available quality assurance measures
The quality management strategy does not outline specific goals and formal evaluation indicators to verify the efficiency of quality assurance
Quality assurance processes do not involve students and social partners – they do not get any feedback on their contribution to quality improvement

44. Recommendations from the experts on IQA
When drafting quality assurance systems, insufficient attention is given to ESG
A process of feedback from students is not standardised or is not analysed
The balance between quality control and quality enhancement is insufficient
Absence of a quality culture within an organisation. Quality assurance procedures and all relevant data should be systemised and processed, while the results should be presented to staff, students and social partners for information purposes in a user-friendly format

45. The internal quality assurance systems of HEIs are Self-regulated responsibilities of the higher education institutions, aimed at Continuous improve- ment of quality and achieving academic excellence.