1. 1 QA Focus – Supporting JISC's Digital Library Programmes Quality Assurance For Digital Library Programmes: The QA Focus Approach Brian Kelly Hamish James UKOLNAHDS http://www.ukoln.ac.uk/qa-focus/ This talk describes the work of the JISC- funded QA Focus project and outlines some options for the future This talk describes the work of the JISC- funded QA Focus project and outlines some options for the future

2. 2 QA Focus – Supporting JISC's Digital Library Programmes An Introduction to QA Why Is QA Needed Now? In the past JISC had standards-based philosophy but: Approach of "let a 1,000 flowers bloom" Diversification as no killer application then (it could have been Gopher!) Things are different now: The Web is the killer application There is now a need for interoperability (not just self-contained independent Web sites) JISC Programmes increasingly intended as service development and not 'speculative' research There is a need for publicly-funded services to be seen to provide value for money Introduction

3. 3 QA Focus – Supporting JISC's Digital Library Programmes QA Focus QA Focus: Funded under the JISC 5/99 programme (CI as contact) Covers technical interoperability (digitisation, Web, metadata, software development & service deployment) Proposal for developmental approach agreed (QA Focus won't be responsible for checking project deliverables) Agreement to develop QA methodology which can be applied to other JISC programmes (QA Focus was funded after other 5/99 projects had started) Understanding that this is a new area of work (for both JISC & ourselves) with a steep learning curve Initially provided by ILRT and UKOLN (1.0 FTE) Provided by UKOLN and AHDS since 1 Jan 2003

4. 4 QA Focus – Supporting JISC's Digital Library Programmes Approach Taken Two possible approaches to ensuring compliance with standards and best practices: Enforce Inspect all projects work Strict auditing, with penalties for no-compliance Encourage Developmental: Working with community Explaining reasons for compliance Documenting examples of best practices Providing advice on implementation & monitoring Developing a QA methodology Seeking to embed QA in working practices This may have been initial assumption in ITT The latter approach is felt to be preferable, especially in a HE context

5. 5 QA Focus – Supporting JISC's Digital Library Programmes QA Focus Activities QA Focus activities: Focus groups to identify key concerns (in areas of standards, implementation and service deployment) Surveys to profile approaches, best practices and common problems Focussed advice primarily addressing: Standards and best practices Implementation issues Compliance checking Case studies describing examples of best practices Development of a QA methodology Validation of approach Embedding QA methodology within projects

6. 6 QA Focus – Supporting JISC's Digital Library Programmes Survey Findings Surveys of compliance of 5/99 project entry points helped to profile community and identify common problems and best practices

7. 7 QA Focus – Supporting JISC's Digital Library Programmes Documentation: Advice Advisory briefing documents are being produced These are: Brief, focussed documents Informed by findings of the surveys Over 40 briefing documents have been published to date Advisory briefing documents are being produced These are: Brief, focussed documents Informed by findings of the surveys Over 40 briefing documents have been published to date

8. 8 QA Focus – Supporting JISC's Digital Library Programmes Documentation: Case Studies Case Studies are being commissioned These are: Written by projects themselves (helping to achieve 'buy-in') Describe the solution adopted to a particular problem Include details of lessons learnt – not just a press release! Over 30 published Case Studies are being commissioned These are: Written by projects themselves (helping to achieve 'buy-in') Describe the solution adopted to a particular problem Include details of lessons learnt – not just a press release! Over 30 published

9. 9 QA Focus – Supporting JISC's Digital Library Programmes Surveys For FAIR & X4L Programmes QA Focus now supports JISC's FAIR and X4L programmes We have developed a self- assessment interface for projects The aim is that projects will (a) check and, if necessary, fix their home page; (b) check/fix other key pages and (c) address underlying causes of problems "I have found the webpage very useful for testing the technical compliance and accessibility of our website" "Many thanks for this - what a really useful tool!" http://www.ukoln.ac.uk/qa-focus/surveys/ fair/web-self-assessment/ QA For Web

10. 10 QA Focus – Supporting JISC's Digital Library Programmes Putting it Together We have: Explained why projects should implement standards and best practices (they don't always know) Provide case studies on approaches by projects Described implementation architectures Described various checking tools But how do we ensure projects : Check deliverables for themselves Have systematic checking procedures There is a need to describe: Tools & architectures for projects for check compliance with standards & best practices Embed systematic QA procedures Tools

11. 11 QA Focus – Supporting JISC's Digital Library Programmes Example - Link Checking Tools We should all be familiar with link checking tools. But: Do such tools check all types of links What about links to external style sheet files, JavaScript files, etc.? Are broken links really broken? Are such tools integrated with normal working practices? Tools

12. 12 QA Focus – Supporting JISC's Digital Library Programmes Describing Tools and Approaches As an example of approach which provides easier access to testing tools, see the,tools approach deployed on UKOLN Web site. This approach: Applies to all resources on Web site Covers HTML and CSS validation, link checking and various other tests Some recursive apps (,rvalidate) Easily implemented with single line redirect Approach is documented at

13. 13 QA Focus – Supporting JISC's Digital Library Programmes Promoting Use Of Compliance- Checking Tools As an example of approach which provides easier access to testing tools, see the,tools approach deployed on UKOLN Web site. This approach: Applies to all resources on Web site Covers HTML and CSS validation and various other tests Some recursive apps (,rvalidate ) Easily implemented with single line redirect As an example of approach which provides easier access to testing tools, see the,tools approach deployed on UKOLN Web site. This approach: Applies to all resources on Web site Covers HTML and CSS validation and various other tests Some recursive apps (,rvalidate ) Easily implemented with single line redirect

14. 14 QA Focus – Supporting JISC's Digital Library Programmes Self Assessment Toolkit Further Deliverables We are developing a self-assessment toolkit for projects to use, by individual projects or across project clusters The toolkit consists of: Self-assessment questionnaires Examples of QA procedures Documented examples of use of testing tools Advice on standards and best practices Case studies FAQs …

15. 15 QA Focus – Supporting JISC's Digital Library Programmes Self Assessment Toolkit (1)

16. 16 QA Focus – Supporting JISC's Digital Library Programmes Self Assessment Toolkit (2) The toolkit is also available in paper format, suitable for use in workshops, meetings, etc. This approach will be used in the forthcoming QA workshop Toolkits on Metadata and Service Deployment will be used

17. 17 QA Focus – Supporting JISC's Digital Library Programmes Next Steps – Metadata QA for metadata is at early stages (little documented work found). Thoughts on QA for metadata: SyntaxCheck syntax on embedded DC Cataloguing RulesEnsure projects have appropriate cataloguing rules InteroperabilityEnsure metadata can interoperate with third parties Fitness for purposeEnsure metadata is appropriate for its purpose Change controlEnsure architecture for managing metadata can cope with change control (cf. Exploit Case Study) See for example

18. 18 QA Focus – Supporting JISC's Digital Library Programmes Next Steps – Software QA for software development is mature, but there may be a need to engage in culture change within the sector Thoughts on QA for software: SpecificationProcess for developing systems spec, agreeing spec with stakeholders, etc. MethodologyDocumentation of systems development methodology (cf UML case study) TestingDocumentation of testing methodology DeploymentAwareness of deployment challenges See for example

19. 19 QA Focus – Supporting JISC's Digital Library Programmes Next Steps – Service Deployment QA for service deployment is important to protect investment in project funding (but there is a lack of awareness in some cases). Development ArchitectureEarly public documentation of development architecture Service ArchitectureAwareness of likely service architecture and liaison with service SecurityAwareness of security concerns LicensingLicensing issues Scenarios Software developed using cool open software environment – but JISC Service has no knowledge of environment. Resources digitised, but copyright clearance not documented. Service refuses to take delivery due to liability concerns. Scenarios Software developed using cool open software environment – but JISC Service has no knowledge of environment. Resources digitised, but copyright clearance not documented. Service refuses to take delivery due to liability concerns.

20. 20 QA Focus – Supporting JISC's Digital Library Programmes The QA Focus Methodology The QA Focus methodology which we are advising projects (and JISC) to deploy: Based on well-established QA principles Seeks to be lightweight and achievable Seeks to provide benefits to the projects themselves, as well as (downstream) interoperability benefits to JISC programmes Consists of: Documented policies Procedures for monitoring compliance with policies Audit trails Sharing and collaboration

21. 21 QA Focus – Supporting JISC's Digital Library Programmes Applying QA – Standards Policy:Open standards used where possible. Proprietary formats with published spec and cross-platforms viewers and authoring tools may be acceptable Monitoring:Project manager responsible Documentation:See QA Focus briefings 11, 31, … Exceptions:If open standards not available, immature or costly to implement proprietary standards may be used. The project manager should document such decisions: why proprietary format was used, migration strategies to open formats in future, architecture for migration and indication of costs

22. 22 QA Focus – Supporting JISC's Digital Library Programmes Applying QA – Web Standards Policy:The Web site will use XHTML 1.0 and CSS 2.0 standards Architecture:The Web site will be based on XHTML templates and SSI Monitoring:New and updated pages validated using,validate and,cssvalidate. Every month,rvalidate will be used Exceptions:HTML derived automatically (e.g. Save As HTML in PowerPoint) need not comply with standards. The files will be stored in a standard directory to enable such files to be excluded from checks.

23. 23 QA Focus – Supporting JISC's Digital Library Programmes Applying QA – Accessibility (1) Policy:The Web site will strive to attain WAI A guidelines. Consistent accessibility shortcuts will be used. An accessibility policy will be published. Architecture:The Web site will be based on XHTML templates which comply with WAI A. Monitoring:New and updated pages will be validated using,bobby. A monthly batch checker will be used and audit reports published (to enable any trends to be spotted). Exceptions:A list of permitted exceptions will be provided. Note that University of Edinburgh Library have a useful Web accessibility policy

24. 24 QA Focus – Supporting JISC's Digital Library Programmes Applying QA – Accessibility (2) Policy:The organisation has no accessibility policy Authors are free to implement their own accessibility shortcuts (if at all) Architecture:No centralised policy covering authoring tools or architecture will be provided Monitoring:No monitoring will be carried out Note that SENDA legislation is helping to encourage organisations to develop appropriate policies and procedures

25. 25 QA Focus – Supporting JISC's Digital Library Programmes A Matrix Approach To Selection Strict Compliance With Standards & Best Practices Projects must implement musts in standards and guidelines document Strict policing & penalties for non-compliance cf. NOF-digitise programme Deployment Of A Standards Culture Standards document describes recommended/preferred standards Projects should complete selection matrix and seek approval on choices Selection will reflect issues such as (a) the standard (maturity, complexity, etc.) (b) implementation issues (availability of tools) (c) organisational culture and (d) programme requirements

26. 26 QA Focus – Supporting JISC's Digital Library Programmes Recommendations QA Focus recommendations to JISC will include: Project proposals should describe approaches to selection of standards, technical architecture & QA Once approved initial project report should document: Standards chosen (e.g. see "ESDS Web Standards Policy" case study and "Matrix for Selection of Standards" briefing document) Justification for any deviations from open standards (e.g. see NOF-TAS FAQ) Technical architecture Intended service deployment environment This process is not intended to be bureaucratic: case studies show that some projects are already engaged in such activities

27. 27 QA Focus – Supporting JISC's Digital Library Programmes Validating Our Approach Feedback on our approaches gained by publishing work in peer-reviewed papers / conferences: Developing A Quality Culture For Digital Library Programmes 1, EUNIS 2003: gave an overview of QA Focus work (paper selected for publication in Informatica) Ideology Or Pragmatism? Open Standards And Cultural Heritage Web Sites 1,2, ichim03: described application of QA Focus methodology for selection of standards Deployment Of Quality Assurance Procedures For Digital Library Programmes 1,3, IADIS 2003: described deployment of QA Focus approaches in CDLR, Univ. of Strathclyde A Proposal For Consistent URIs For Checking Compliance With Web Standards 1,3, IADIS 2003 1Written by UKOLN and/or AHDS staff 2Co-authored with TechDis staff 3Co-authored with CDLR staff

28. 28 QA Focus – Supporting JISC's Digital Library Programmes Thoughts On Timeliness of QA Early adopters Hype Despondency Realism & service deployment QA implemented Maturity Software Digitisation Web Metadata Web Services Semantic Web

29. 29 QA Focus – Supporting JISC's Digital Library Programmes What Of The Future? (1) After The QA Focus Funding Ceases QA Focus Deliverables The QA Focus deliverables (documents, papers, tools, etc.) will continue to be hosted at a stable URL for an extended period (3-10 years) QA Focus Recommendations Recommendations to JISC may be implemented in future JISC programmes and technical advice, etc. carried out by Programme Managers Recommendations may be ignored and (a) alternative approach deployed (b) QA issues not formally addressed

30. 30 QA Focus – Supporting JISC's Digital Library Programmes What Of The Future? (2) Option 2 Funding Continued QA Focus Deliverables The QA Focus deliverables (documents, papers, tools, etc.) will be maintained, new areas addressed, existing areas addressed in more depth, etc. Remit extends beyond 5/99, FAIR & X4L QA Focus Recommendations Recommendations to JISC implemented in future JISC programmes Technical advice, involvement with programmes and projects, etc. carried out by QA Focus team working closely with Programme Managers

31. 31 QA Focus – Supporting JISC's Digital Library Programmes What Of The Future? (3) Option 3 Alternative Approaches Hardline QA JISC wishes to implement a hardline approach to QA Remit of MU extended to include technical aspects of project deliverables No QA JISC decides that implementing a formal QA infrastructure is inappropriate Projects left to 'do their own thing' But what are the implications of these contrasting approaches?

32. 32 QA Focus – Supporting JISC's Digital Library Programmes The Big Picture The QA Focus approach could potentially be applied to a variety of audiences: QA for current and new IE programmes QA for all JISC programmes QA for JISC QA for JISC RSCs QA for JISC Services QA for UK HEIs and FEs QA for CIE (and links with other sectors) Clearly this raises many issues!

33. 33 QA Focus – Supporting JISC's Digital Library Programmes Conclusions We have described: The approaches to QA which have been taken by the QA Focus project The resources which have been developed for the community by QA Focus A systematic methodology for QA Options for future work in this area