1. Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna 15 and 16 December 2008

2. Relevance, impact and evaluation Various functions of evaluation  Judgment about Quality, both scientific and societal  Accountability to university, to government, to society  Improvement of research quality and impact Various problems  Functions get mixed in evaluations, but with different weight  Some seem to be more obvious than others  Criteria and indicators slanted towards scientific quality and to practices of natural and biomedical sciences  Different actors have different expectations Peer review under pressure, evaluation procedures too

3. Evaluating Research in Context (www.eric-project.nl) endorsed by * Dutch Academy, * National research council (NWO), * Association of Dutch universities, * Council of the professional schools, * Rathenau Institute (Science Systems assessment) * QANU, * Ministry of Education and Sciences

4. ERiC: Development of alternative evaluation methods WHY ERiC  to inform boards / government about societal relevance of research (accountability and Lisbon ambitions)  to help research areas that are not happy with dominant evaluation approaches : humanities, social sciences, technical sciences, health research, MIT, etc.

5. Core problem  The world is changing  Science (academic) is changing into research (MIT) (Latour)  Evaluation is not yet changing, or at least still dominated by criteria and indicators that used to work in the old situation

6. methodological development  RMW: the societal impact of health research [2002]  SWR / RGW: judging research on its merits [2005]  ERiC: evaluating research in context [2005, 2007]  AWT: alfa stralen (humanities shine)  disciplinary developments: health research (RGO / UMCs), development research (CERES); law (disciplinary committee)  pay back, UK research councils, Scandinavian countries, etc.  examples from outside Europe: USA, Australia, Canada

7. “Judging research on its merits” [KNAW/SWR/RGW 2005]  Collaboration of two Academy Councils, social sciences and humanities  Dissatisfied with current evaluation systems mainly focusing on SCI (high impact journals)  Two problems: 1. doesn’t necessarily fit research communication patterns in soc sc and hum; 2. doesn’t value societal output which is important for a lot of fields  Search for more fitting evaluation schemes

8. Necessary conditions for better evaluations  Self evaluation focus on both scientific reputation and communication, and also on interaction with other audiences  List of target groups: peers, students, professionals, policy makers, business, broader public  List of indicators per target group: publications, citations, but also text books, reviews, grants from policy, collaborations with business, professionals, awards, popular publications, etc.  Benchmarking is critical process

9. Goals of ERiC  stimulate debate about alternative methods of evaluation, fit for all disciplines regarding the value of research for society  develop methods to do such evaluations  website www.eric-project.nl

10. how does ERiC work?  Context group as a central unit  publications  guidelines  workshops, national and international  pilot studies : agricultural research, pharmaceutical research, architecture, law, engineering  European project : SIAMPI  website : www.eric-project.nl

11. Results so far  Working definition of Societal value: oAs a broad concept : societal relevance oAs a target-oriented concept : societal impact oAs a mainly economic concept : valorization these three have different consequences for evaluation  Assessment approach fit for all disciplines:  based on an analysis of the mission orientation of a research group oscientific community oprofessional sector oIndustry opolicy-oriented osociety at large  Test method

12. Test model: 4 steps 1.Reflection on mission orientation, self evaluation report 2.Empirical stage, find criteria and indicators for quality and relevance 3.Consultation with researchers and stakeholders, presentation of findings 4.Feed back and forward look

13. Step 2 : productive interactions  Identify social domains in context (policy, industry, society at large)  Analyse the 4 main interaction channels between research and context: texts, people, artifacts, money  Feed back to research community and stakeholders [workshops]

14. example of evaluation of societal quality – radar graph

15. example REPP – table graph Science, certified knowledge relative citation impact -- productivity scientific publications ++** (1 international visibility and collaborations = representation in editorial boards ++ invited lectures ++ Industry, market non-academic/commercial citing environment ++ productivity professional publications ++* (1 involvement in industry/market - advisory and expert roles in commercial domain -- editorships professional journal ++** Policy, societal involvement in policy domain + memberships and expert roles in governmental bodies ++ memberships of societal organisations: advisory/ education ++* production of public goods + additional grants from policy +

16. example of evaluation of societal quality – radar graph

17. ERiC and the humanities  Express the relations of humanities research with society in a meaningful way  Help find criteria and indicators that might help to evaluate that  Bring together the expertise of both stakeholders and researchers  Stimulate them to find consensus about how to evaluate

18. What can humanities do?  don’t try to emulate other fields  try and find consensus as a field about indicators  convince policy makers and managers that you are capable of defining indicators that are robust and fit for your field

19. Netherlands France, NL, EU NL, UK Spain, UK Health Nanotechnology ICT Social sciences and humanities ERiC SIAMPI www.ERiC-project.nl

20. focus in human sciences 1.Describe the content, (societal) objectives, organization and evaluation practices of the program/projects involved in the pilot. 2.Identify research strategies and policies aiming at measuring the scientific and social impact of the program/projects in the various relevant dimensions. 3.Discuss the findings with the stakeholders related to this case, evaluating the validity, reliability and relevance of the results. 4.Advise on further development of methods for the assessment of social impact. Analysis of strength, weaknesses and possible improvements for the method to be developed.

21. ERiC and SIAMPI and the Humanities 1.ERiC : cases in the Netherlands, but perhaps abroad 2.SIAMPI : dissemination of results through comments, conferences, joint workshops