Presentation is loading. Please wait.

Presentation is loading. Please wait.

Is success divisible? How to evaluate success successfully?! New Frontiers In Evaluation Session F Talking about Success April 25 th 06 - 9.40 – 12.30.

Similar presentations


Presentation on theme: "Is success divisible? How to evaluate success successfully?! New Frontiers In Evaluation Session F Talking about Success April 25 th 06 - 9.40 – 12.30."— Presentation transcript:

1 Is success divisible? How to evaluate success successfully?! New Frontiers In Evaluation Session F Talking about Success April 25 th 06 - 9.40 – 12.30 a.m. Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 Institute for Researchinformation and Quality Assurance Stefan Hornbostel Saskia Heise IFQ Institut für Forschungsinformation und Qualitätssicherung Institute for Researchinformation and Quality Assurance Godesberger Allee 90 D-53175 Bonn www.forschungsinfo.de

2 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 IFQ-Project Final Reports The work invested in final reports receives little to no appreciation: - for the scientists only the findings and publications from a project are of importance and not writing a final report - for the experts the judgment about the funded project is given - from the viewpoint of the funding agency a project is finished with the end of its promotion. We suggest that the pieces of information included in a final report should be suitable for the monitoring of research activity. On the basis of our project, we would like to investigate whether final reports - in particular the documented output of a project - could be used as an evaluative tool in the DFG research funding.

3 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 IFQ-Project Final Reports Targets of our project Final Reports" are (1)to build up a research monitoring system (2)to develop an information tool which provides web-access to the findings

4 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) Guidelines for final reports (Manual) 1.General Information (DFG-reference-number, applicants, project leader, institution, topic, promotion period, publications) 2.Research Activities and Findings (exploratory questions and target, conducted work – in particular discrepancies, scientific failures – findings, application, connections, usability, patents, industrial co-operations, co-operation partners, project collaborators, diploma, PhD-theses, postdoctoral qualifications) 3.Summary (generally understandable presentation, unexpected results in the developing process, dissemination of information outside the scintific community - possibly press coverage)

5 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 References for drafting of an expert's report on final reports Short Expertise 1.Comment on the form of the report (outline, layout) 2.Comment on the content (level of achievement, duration, methodology) 3.Evaluation of the findings (proportionate awareness and promotion, quality of publications, additional requirements) 4.Additional Comments

6 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 In Final Reports of DFG-funded projects included information, Heise 2006

7 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 What relevant information could be asked from the project leaders and how?, Färkkilä 2004

8 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.06 Which of the following purposes of research evaluation apply to your organisation? N = 15 European Funding Agencies

9 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.06 What does success mean from the viewpoint of a funding agency?

10 Institut für Forschungsinformation und Qualitätssicherung 25.04.06

11 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.06 Co-authorship in selected years - SCI papers from all fields Share of single- authors papers Mean number of Co- authors 198024,8 %2,64 199015,7 %3,34 200010,7 %4,16 Source: Glänzel. W. & Schubert, A. (2004): Analyzing Scientific Networks Through Co-Authorship. In: Moed, H.F. et al.: Handbook of Quantitative Science and Technology Research, 257-276

12 Institut für Forschungsinformation und Qualitätssicherung 25.04.06 Fractional CountingComplete Counting Paternity Test Shareholder (author, co-author, co-writer sub-author, contributor, hyper- author) Creatership Level: Individual Scientist (multi-authorship)

13 Institut für Forschungsinformation und Qualitätssicherung 25.04.06 Level: Project

14 Institut für Forschungsinformation und Qualitätssicherung 25.04.06 Furthermore, many sources of research funding expect researchers to acknowledge any support that contributed to the published work. Just as citation indexing proved to be an important tool for evaluating research contributions, we argue that acknowledgements can be considered as a metric parallel to citations in the academic audit process. C. Lee Giles and Isaac G. Councill: Who gets acknowledged: Measuring scientific contributions through automatic acknowledgement indexing. In: Proceedings of the National Academy of Sciences 101(51) pp. 17599-17604, Dec. 21, 2004. Of 335,000 unique research documents within the CiteSeer computer science archive 188,052 were found to contain acknowledgements (roughly 56%)

15 Institut für Forschungsinformation und Qualitätssicherung 25.04.06 C. Lee Giles and Isaac G. Councill: Who gets acknowledged: Measuring scientific contributions through automatic acknowledgement indexing. In: Proceedings of the National Academy of Sciences 101(51) pp. 17599-17604, Dec. 21, 2004. both the German Science Foundation (Deutsche Forschungsgemeinschaft) and the United Kingdom Engineering and Physical Sciences Research Council (EPSRC) display a steady upward trend in the proportion of acknowledgements received each year during the 1990s while the Office of Naval Research and IBM slowly become overshadowed by other entities over the decade.

16 Institut für Forschungsinformation und Qualitätssicherung 25.04.06 Multidimensional Access Methods - Gaede, Günther (1997)Multidimensional Access Methods - Gaede, Günther (1997) (Correct) (253 citations) partially supported by the German Research Society (DFG/SFB 373) and by the ESPRIT Working Group CONTESSA www.wiwi.hu-berlin.de/~gaede/survey.rev.ps.Z Efficient PRAM Simulation on a Distributed Memory Machine - Karp, Luby, der Heide (1992) (Correct) (72 citations) Science Institute at Berkeley supported in part by DFG-Forschergruppe "Effiziente Nutzung massiv ftp.uni-paderborn.de/doc/techreports/Informatik/tr-ri-93-134.ps.Z(Correct)(253 citations) Efficient PRAM Simulation on a Distributed Memory Machine - Karp, Luby, der Heide (1992)(Correct)(72 citations) Decoding Choice Encodings - Nestmann, Pierce (1996)Decoding Choice Encodings - Nestmann, Pierce (1996) (Correct) (52 citations) supported by the DFG, Sonderforschungsbereich 182, project C2, and by www.cs.auc.dk/~uwe/self/doc/concur96.ps.gz(Correct)(52 citations) Statistical Models for Co-occurrence Data - Hofmann, Puzicha (1998)Statistical Models for Co-occurrence Data - Hofmann, Puzicha (1998) (Correct) (24 citations) was supported by the German Research Foundation (DFG) under grant #BU 914/3-1. 1 Introduction The publications.ai.mit.edu/ai-publications/1500-1999/AIM-1625.ps(Correct)(24 citations) Multiresolution Analysis of Arbitrary MeshesMultiresolution Analysis of Arbitrary Meshes (222 Citations) Matthias Eck, Tony DeRose, Tom Duchamp, Hugues Hoppe, Michael Lounsbery, Werner Stuetzle (1995) This work was supported in part by a postdoctoral fellowship for the lead author (Eck) from the German Research Foundation (DFG), Alias Research Inc., Microsoft Corp., and the National Science Foundation under grants On Evaluating Decision Procedures for Modal LogicOn Evaluating Decision Procedures for Modal Logic (58 Citations) Ullrich Hustadt, Renate A. Schmidt (1997) We thank Christoph Weidenbach and Andreas Nonnengart for their critical comments. The work of the second author is supported by the TraLos- Project funded by the DFG http://www.ag2.mpi-sb.mpg.de/~schmidt/publications/MPI-I-97-2-003.ps.gz An asymptotically optimal multiversion B-treeAn asymptotically optimal multiversion B-tree (57 Citations) …….Seeger, Peter Widmayer (1996)We want to thank an anonymous referee for an extraordinary effort and thorough discussion that led to a great improvement in the presentation of the paper. This work was partially supported by grants ESPRIT 6881 of the European Community and Wi810/2--5 of the Deutsche Forschungsgemeinschaft DFG http://medoc.springer.de:9999/Journals/vldb/tocs/../papers/6005004/60050264.ps.gz810/2--5

17 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 Within the preparations of our project Final Reports we held explorative interviews (by telephone, average 20 min. duration) with DFG-applicants and asked them, how they would appreciate assignments between project results and producers. Experts from the disciplines chemistry, engineering and educational science participated in the study. What does success mean from the viewpoint of a recipient of a grant?

18 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 Chemistry The interviewed chemists generally find it possible to assign research results to individual projects. With the exception that in interdisciplinary projects or networking it comes to cross-linking and project overlaps, stimulation of projects from another context and therefore no one-to-one allocation of ideas is possible. Though they consider it possible to assign authors to a project context and to published findings. Without exception the asked applicants from chemistry acknowledge the third- party-funding organisation in all publications that emerge from the project. In the case of multiple funding, they name every involved funding agency.

19 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 Engineering Scientific engineers consider it difficult to relate project output exactly to one project. Results emerge from interal co-operations with other professorships, interactions between scientists that may be relevant for different projects. While relatively unquestionably publications can be appropriate to authors, the allocation from findings to project contexts is more difficult, due to parallel projects, and is only later reconstructed. Engineers co-operate with scientists and industry, co-operation is sometimes agreed only for the duration of a certain project, sometimes for longer periods. Though in the end, all asked scientific engineers acknowledge the third-party- funding organisation in their publications.

20 Institut für Forschungsinformation und Qualitätssicherung Hornbostel/Heise 25.04.2006 Educational Science The asked scientists consider it possible to assign project output and publications to a concrete project context. It may happen that publications from a project context emerge years later (5-10 years). Publications can be related to persons, results are always clearly recognizable. All asked persons acknowledge the third-party-funding promoter. In educational science this is considered very important – funding by the DFG stands for quality. In educational science co-operation exists for longer periods of and time is usually concretized in preposition of a project framework.

21 Institut für Forschungsinformation und Qualitätssicherung 25.04.06 Increasing Pressure Peer Assessment Grant / Project Evaluation Funding Agencies call for Acknowledgments (as a kind of Property Right) Scientists feel obliged to assign outcome to projects Use of acknowledgments as indicators

22 Institut für Forschungsinformation und Qualitätssicherung 25.04.06 Success has many fathers, but failure is an orphan The relation between project and project outcome is a social construction to the same degree as relationship between author and publication What we measure is the construction set by scientists, not the causal dependency The evaluation window is not identical with the term of a project Project outcome is more than publications (Phds, networks, transfer, patents, dissemination of information outside the scientific community …. – some of them hard to meassure) Therefore success-indicators (like citations etc.) should be combined with peer assessment of final reports Objectives of systematic use of final reports are - information about programme performance - information for applicants - information for peers about former performance of applicants Lessons learned:

23 Institut für Forschungsinformation und Qualitätssicherung 25.04.06 Thank you !


Download ppt "Is success divisible? How to evaluate success successfully?! New Frontiers In Evaluation Session F Talking about Success April 25 th 06 - 9.40 – 12.30."

Similar presentations


Ads by Google