1. Models of Evaluation For Research Proposals in Turkey Prof. Dr. Omer CEBECI Vice President – Funding, TUBITAK (The Scientific and Technical Research Council of Turkey) & S. GENC, A. KERC, H. KARATAS, A. FEYZIOGLU, F. COSKUN, O. OZPEYNIRCI, B. DIKMEN, G. KOZANOGLU, H. GULER & O. KARA

2. SUMMARY Research funding in Turkey, TUBITAK & recent boost Three-dimensional evalution used by TUBITAK for the evaluation/selection of research project proposals grouped under three categories: (1) curiosity driven academic research, and (2) customer driven applied research, both in universities and research institutions, and (3) technological and innovation driven research conducted by the private industry, plus (4) research equipment & infrastructure. Details of the “Phrase-anchored rating scale”

3. Actors of Turkish National Science and Technology System (policy makers) BTYK- Supreme Council of Science and Technology ME-Ministry of Education MIT-Ministry of Industry and Trade TUBITAK-The Scientific and Technical Research Council of Turkey HEC-Higher Education Council (YÖK) The Turkish Academy of Sciences (TÜBA) KOSGEB- Small and Medium Sized Industry and Development Organizations TAEK- The Turkish Atomic Energy Commission TOBB-Union of Chambers and Commodity Exchange of Turkey DTM-Foreign Trade and Treasury TTGV- Technology Development Foundation of Turkey DPT- State Planning Organization TÜRKAK- Accreditation Board

4. TUBITAK National Research Centers and Institutes  Marmara Research Center (MAM)  Institute of Energy  Chemistry and Environment Institute  Food Institute  Materials Institute  Information Technologies Research Institute  Earth and Marine Sciences Research Institute  Information Technologies and Electronics Research Institute (BİLTEN)  National Electronics and Criptology Research Institute (UEKAE)  Defence Industry Research and Development Institute(SAGE)  Basic Science Research Institute  National Academic Network Center  Genetic Engineering and Biotechnology Research Institute  National Metrology Institute  Turkish Industry Management Institute  National Observatory Actors of Turkish National Science and Technology System (science and research performers) Turkish State Universities Turkish Industry Other National Research Centers and Institutes Turkish Private Universities and Research Centers

5. President of Republic Prime Minister BTYK YÖK TÜBİTAK STB BTYK- Supreme Council of Science and Technology STB- Minister of Industry and Trade MEB- Ministry of National Education TÜBİTAK- The Scientific and Technological Research Council of Turkey YÖK- Higher Education Council (YÖK) TÜBA- The Turkish Academy of Sciences (TÜBA) DPT- State Planning Organization TÜRKAK- Accreditation Board MEB Universities DPTTÜBA TÜRKAK TPE KOSGEB TSE TTGV R&D Institutions TPE- Turkish Patent Institute TSE- Turkish Standards Institution DIE- State Institute of Statistics DTM- Undersecretariat of the Prime Minister for Foreign Trade TOBB-Union of Chambers and Commodity Exchange of Turkey KOSGEB- Small and Medium Industry Development Organizations TTGV- Technology Development Foundation of Turkey DTM TOBB DIE Turkish National Science, Technology and Innovation System

6. National S&T Indicators EU-15 Values (2000-2001): Number of triadic patents per million population:36 Number of papers per million population:822 GERD per capita population:467. 6 (PPP $) GERD as a percentage of GDP:1,89 Total R&D personnel per 1000 employment: 10.4 0 0,5 1 1,5 2 2,5 Number of triadic patents per million population Number of papers per million population GERD per capita population GERD as a percentage of GDP Total R&D personnel per 1000 employment EU-15 Japan US Germany

7. 0,5 1 GERD as a percentage of GDP Spain EU-15 Hungary Turkey National S&T Indicators Total R&D personnel per 1000 employment Number of triadic patents per million population Number of papers per million population GERD per capita population

8. Scientific Publications

9. Knowledge as a Tool Knowledge is a tool that can be used for a variety of social objectives, including:  Meeting Basic Human Needs  Increasing Safety & Security  Improving the Quality of Life  Economic Growth and Development

10. 100.000 $Communication Satellite 4.000.000 $Microprocessor (Si) Chip 10.000 $Combat aircraft 3.000 $Fiberoptical Cable 100 – 1.000 $Passenger Aircraft 10 – 100 $Automobile 1,5 $Aluminium 50 centSteel 5 centCement 1 centConcrete Knowledge - R&D Cost per Kg

11. Share of TUBITAK in TR Fiscal Budget

12. Research Proposals *from universities and research institutions: 1) curiosity driven academic research 2) customer driven applied research *from private industry: 3) technological & innovation driven research

13. TUBITAK Research Project Proposals Science and Technology Project Proposals 0 500 1000 1500 2000 2500 3000 199920002001200220032004 Years Number of Proposal Career International Sector Units Infrastructure Research

14. Running Research Projects

15. EVALUATION & MEASUREMENT

16. TARİHTEN BUGÜNE SAĞLIK ve ÖLÇME Ateş ve nabız – 1625 (Santario ve Galileo) Torricelli - 1643 Tansiyon - 1733 (Hales) 1896 - Riva-Rocci Korotkov - 1905 Ateş - nabız - tansiyon ölçüm ve kayıtları ile hasta izleme - 1920

17. ÖLÇME ve DEĞERLENDİRME “ Ölçemediğinizi anlıyamazsınız ! ” Lord Kelvin (1824 - 1907) (If you can’t measure it - you don’t understand it) “Her hesaba katılması gerekenin sayılır olması gerekmez ! “ Albert Einstein (1879 - 1955) (Not everything that can be counted counts & not everything that counts can be counted)

18. Çıktı mı ? Sonuç mu ? Etki mi? Çıktıların sonucu olarak gerçekleşen ETKİ !

19. OUTPUT – OUTCOME – IMPACT IMPACTS achieved as consequences of the OUTCOMES resulting from the realisation of the OUTPUTS

20. OUTPUT – OUTCOME – IMPACT IMPACTS achieved as consequences of the OUTCOMES resulting from the realisation of the OUTPUTS ( My eye jelly project )

21. Intended / Expected / Desired IMPACT :... in full control of my mandate... Unintended / Unexpected / Unwanted BUT ACTUAL REAL IMPACT :... under strict control of my wife !..

22. Evaluation Criteria three-dimensional evaluation criteria developed in collaboration with researchers & reviewers

23. 1) Curiosity Driven Academic Research - the three dimensions - 1.intellectual/scientific/professional merit 2.expected impact of the anticipated outcome 3.achievability of the research with the proposed research team, equipment/facilities and methods

24. 1.research and development merit 2.expected impact of the anticipated outcome 3.achievability of the research with the proposed research team, equipment/facilities and methods 2) Customer Driven Applied Research - the three dimensions -

25. 1.technological level of the research 2.innovative level of the product/outcome 3.feasibility of the process 3) Technological & Innovation Driven Research - the three dimensions -

26. Evaluation Process all three-criteria are given equal weights Phrase-Anchored Rating Scale versus Likert Scale sub-criteria phrases describing *very competitive *competitive *not-competitive features of the proposal

27. Very competitive *scientifically and professionally outstanding & very well justified project & *points to an opportunity for a major contribution to the advancement of the knowledge and/or to the resolution of a problem of practical importance

28. Competitive *scientifically and professionally competent and justified proposal which will make a contribution to the advancement of knowledge &/or the resolution of a problem of practical value & therefore *support is suggested if funds are available

29. Not competitive *work routine in character *scientifically and professionally unsatisfactory and poorly organized

30. Evaluation Process evaluate 10-15 proposals by referring to the sub-criteria phrases a meeting for a final verdict Panels 5-8 Individual Reviewers

31. Curiosity / Merit = V. Comp. Outcomes have high potentials for publication in journals or books listed in international indexes The originality of the work has been supported by extensive and critical literature survey Hypothesis for evaluating the research topic is very well defined Explanation and analysis of the expected outcomes reveal the superiority of the work in comparison to the existing science and technology.

32. Curiosity / Merit = Not comp. Scientific consistency and the rationale of the research are not clearly explained. A clear scientific / technological question is not put forward. Project is more like an investigation / data collection / routine work. References provided include similar studies & literature survey does not point out the basis / importance of the project. The research is based on unreliable data and hypothesis.

33. Curiosity / Impact = V. Comp. Very important in terms of sustainable development of the country /very likely to be implemented to find solutions for the problems of society Very likely to be employed in different scientific & technological fields Very likely to generate new projects Commercialization potential of the outcomes is very high The project is supported by international, national or industrial sources.

34. Curiosity / Impact = Not comp. Potential for adding value to science and technology is low Subject of the project is not among the priorities of the country Not likely to result in intellectual property worth-protecting.

35. Curiosity / Achiev. = V. Comp. Project Team The team is experienced in national / international projects related with same/similar subject They have experience as advisors /authors / referees / editors / book authors The can allocate enough time for the project Competencies / responsibilities / roles of the team members are well defined and adequate End users of the project outcomes are also members of the team.

36. Curiosity / Achiev. = Not Comp. Project Team The team is not experienced in conducting projects of this size The knowledge and awareness of the team is not sufficient They do not have important publications in the subject of the project Some of the team members are irrelevant for the project Essential competencies are lacking

37. Curiosity / Achieve. = V. Comp. Infrastructure Infrastructure of the institution is very adequate for the project Additional equipment requested within the scope of the project is very compatible with the existing infrastructure and the project Existing sources / equipment are used rather than purchasing new sources Requested equipment can also be used in other / future projects

38. Curiosity / Achieve. = Not Comp. Infrastructure Infrastructure of the institution is not adequate for the project, unless supported with major equipment Equipment requested within the scope of the project is not compatible with the existing infrastructure and the project

39. Curiosity / Achieve. = V. Comp. Methodology Approach / methodology are very well designed to reach the target Methodology is correct and well-defined; and standard methods and literature are cited Preliminary experiments have been conducted to rationalize the hypothesis Alternatives (plan B) have been considered if difficulties are encountered

40. Curiosity / Achieve. = Not Comp. Methodology Methodology is not adequate to reach the target Relations between the experiments and hypothesis are not well defined Methodology is not explained with a common scientific basis Possible problems and limitations are not considered Statistical analytical requirements are not considered

41. Curiosity / Achieve. = V. Comp. Timeline Proposed period & time schedule are realistic Budget Proposed budget is realistic and well-justified Project is also supported by other institutions

42. Curiosity / Achieve. = Not Comp. Timeline Proposed period and time schedule are not synchronized Time schedule is not adequate Budget Budget is not well-define and requested amount is too low / high There is no possibility of support from other institutions

43. Customer / Merit = V. Comp. Aims to develop a national / international novel technology (methodology, system, product, process / technique) Brings comparable superiority to the existing system Outcome will be a technology to be protected under the intellectual property rights Work consists of a scientific and rationale approach An interdisciplinary project with the collaboration of the Public- Private Industry- University-Research Institutions

44. Customer / Merit = Not comp. Literature survey and market survey are not satisfactory, work is based on unreliable data No scientific / technological rationale and integrity in the project It is a study that had been done before in the country Aim, objective and motivation are not clear Outcome is not qualified as “applicable/usable” Not related with a real need / problem

45. Technological Level = V. Comp. Technology / product developed aims to fill a gap in the existing technology or replace the existing technology within the following 3 years Has an interdisciplinary approach to solve more than one problem Rationale of the R&D is well established (theoretical / analytical / experimental) Added value of the anticipated outcome of the R&D project is considered Has contribution to increase R&D staff A doctorate / masters study is incorporated within the project and the outcomes have potentials to be published in national/international journals.

46. Technological Level = Not comp. Literature survey and market survey are not adequate No scientific/technological advance and integrity in the project R&D rationale (analytical and /or experimental) is not adequate Aim, objective and motivation are not clear.- Procedure that has been used will not provide a change / improvement in the methodology and technology

47. Innovation Level = V. comp. Outcomes may lead to spin-off company Outcomes include production standards and technical specifications Outcomes will affect other sectors as well Outcomes very likely to increase the export capacity of the country Very likely to be a basis for generating new projects / products

48. Innovation Level = Not comp. Outcome is not qualified as “applicable/usable” Period of usefulness is very limited or not effective Does not provide new areas of work and as a result does not increase employment Expenditure for the R&D study is far beyond the expected economical benefit of the product Support provided by the private organization is not sufficient.

49. Comments Reviewers (hundreds): The model reduced the, –burden of the review process on them, & – subjectivity and variability of the opinions of the individuals Convenience of selecting from a comprehensive list of phrases as well as the freedom of offering their own judgments Researchers (thousands): Sub-criteria phrases to be very instrumental in guiding the development of their proposals

50. Thank you Hundreds of scientists, researchers and reviewers participated in workshops for the development of the criteria and phrases listed in the tables. Their contributions are gratefully acknowledged.