4 th November 2005 JISC VRE Advisory Board Meeting Formative evaluation workshop: initial feedback and discussion John Kelleher and Kerstin Junge Tavistock Institute, London

Formative Evaluation of JISC VRE Nov Introduction This workshop Presentation and discussion a.Brief introduction to the Tavistock and its ways of working (10 to 15 minutes) b.The formative evaluation approach and progress to date (15/20 ‘’ ) c.Initial findings: some characteristics of the programme as implemented (20/25 ‘’ )

Formative Evaluation of JISC VRE Nov Introduction This workshop Workshop d.Emergent issues (20/25 minutes) e.Evaluation concerns and foci (20/25 ‘’ ) f.Implications and consequences (20/25 ‘’ )

Formative Evaluation of JISC VRE Nov The Tavistock Institute a. The Tavistock and its ways of working Research and consultancy organisation ‘supporting and learning from change and innovation’ Bias towards Action Research Formative and other evaluations of eLib and several other JISC programmes and activities

Formative Evaluation of JISC VRE Nov Formative evaluation approach and progress to date b. Formative evaluation Learning from innovation Timely ongoing feedback to support steering and development Part of the Action Research cycle (plan – act – evaluate - plan - …)

Formative Evaluation of JISC VRE Nov Formative evaluation approach and progress to date This formative evaluation Phase I Documentary and literature review Key interviews Project visits (5 so far, completion by late Nov.) Policy map (in progress) Interim report (December, JCSR Feb.)

Formative Evaluation of JISC VRE Nov Formative evaluation approach and progress to date This formative evaluation Phase II Planning Project and programme support: individual projects; sets of projects; all projects; programme User needs, self evaluation, institutional embedding, collaboration, …

Formative Evaluation of JISC VRE Nov Initial findings c. Some characteristics of the programme Project types Technically orientated – user orientated Development – research Multiple institutions – single institutions Arts & humanities – natural sciences Single discipline – multiple disciplines Users imagined – users investigated

Formative Evaluation of JISC VRE Nov Initial findings Some characteristics of the programme Orientations and approaches Tools more than technologies Implicit rather than explicit user models Institutional settings assumed rather than addressed ‘Revolutionaries’ versus ‘evolutionists’ ‘True believers’ (in VREs) v ‘agnostics’ (believers in e-research)

Formative Evaluation of JISC VRE Nov Initial findings Some characteristics of the programme Memberships Presence of leading edge users in teams Absence of institutional decision makers in projects Sparse organisational and programme design and change expertise

Formative Evaluation of JISC VRE Nov Initial findings Some characteristics of the programme Initial project issues arising User engagement Environmental embedding Intra-programme exchange and learning Exit and exploitation

Formative Evaluation of JISC VRE Nov Emergent issues d. What do WE mean by … ‘VRE’? ‘Users’? ‘Embedding’? ‘Products’? ‘Learning’? ‘Policy’?

Formative Evaluation of JISC VRE Nov Evaluation concerns and foci e. Evaluation concerns and foci? Social, organisational and institutional assumptions Strategies of engagement … of users, of institutional decision makers Emergent agendas for action

Formative Evaluation of JISC VRE Nov Implications and consequences f. Implications and consequences? For the formative evaluation design … For the programme (VRE1) … For the next programme (VRE2) …