1. Proposal Writing & The Refereeing Process
Title
Proposal Writing & The Refereeing Process
Class 4
Class 4, 27/11/2006
FirstName LastName – Activity / Group

2. The Refereeing Process
Title
The Refereeing Process
Class 4, 27/11/2006
FirstName LastName – Activity / Group

3. Contents Introduction Peer review process Journals Conferences
Title
Introduction
Peer review process
Journals
Conferences
Research programmes
The tasks of a referee
Reviewing a research paper
Preparing the referee report & recommendations
Evaluating a research proposal
Acting as an editor or program chairperson
How to become a referee?
Final words
Class 4, 27/11/2006
FirstName LastName – Activity / Group

4. Disclaimer There is no fixed mechanism for refereeing
Title
There is no fixed mechanism for refereeing
There are simple rules that help transforming a review in a constructive document
In time you will develop your own style of refereeing
Class 4, 27/11/2006
FirstName LastName – Activity / Group

5. Introduction
Title
A scientific paper is expected to provide a sufficient contribution to the knowledge base of its field
Number of scientific papers and articles (2000): > (ISI)
About 50% in the fields of science and technology
The number of papers and articles submitted for publication is much larger
refereeing process selects the ones to be published
Examples of acceptance rates after refereeing:
Journals: ~10-20% (large variance)
Conferences: ~10-50%
Workshops: ~30%-90%
Refereeing is also used in selecting research projects to be funded
Class 4, 27/11/2006
FirstName LastName – Activity / Group

6. Introduction What is a sufficient contribution?
Title
What is a sufficient contribution?
new result, theoretical or experimental
new insight
novel synthesis of ideas
useful survey
useful tutorial
What is not a sufficient contribution
badly written
erroneous data
MPI = Minimum Publishable Increment depends on the forum
Class 4, 27/11/2006
FirstName LastName – Activity / Group

7. Peer review process
Title
Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution
novelty, significance, correctness, readability
Refereeing is public service to the scientific community
professional obligation,
carried out on volunteer basis
requires high expertise
helps in improving one’s own expertise
ensures the integrity of science
Class 4, 27/11/2006
FirstName LastName – Activity / Group

8. Peer review process of a journal
Title
submission
editor
author
publish
accept
reject
revise
selection of associate editor
reviews
recommendations
associate
editors
referees
selection of referees
checking of revised papers
Class 4, 27/11/2006
FirstName LastName – Activity / Group

9. Peer review process of a conference
Title
submission
program
committee
program chair
author
accept/
reject/
accept with revisions
selection of the referees
checking of revisions
accept/reject/minor revision recommendations
referees
extra referees
Class 4, 27/11/2006
FirstName LastName – Activity / Group

10. Peer review process of a workshop
Title
submit
program
committee
program chair
author
accept/
reject
refereeing
checking of
revisions
extra referees
Class 4, 27/11/2006
FirstName LastName – Activity / Group

11. Peer review process of a research programme
Title
submission
steering
committee
proposer
accept with partial funding/
reject
referees
Notice: not representative of all research programmes
Class 4, 27/11/2006
FirstName LastName – Activity / Group

12. The tasks of a referee
Title
The reviewer grades a paper based on its novelty, significance,correctness, and readability
In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly
Both positive and negative findings are summarized in a referee report
Confidential part only for the editor/program committee: Information that could reveal the identity of the reviewer or in minor conflicts of interest
non-confidential part for the author/program committee
Learn from the other reviews, if they are sent to you after the process
Class 4, 27/11/2006
FirstName LastName – Activity / Group

13. Why do it? Several reasons
Title
Several reasons
Enhance reputation (with editor/prog. committee)
Expedites processing of your own papers
Get on editorial board or program committee
Good practice
Increase your own critical appraisal ability
Your papers become better
Sometimes it gets preferential treatment for your papers
… but refereeing means more work!
Class 4, 27/11/2006
FirstName LastName – Activity / Group

14. Consideration Most reviews have strict deadlines
Title
Most reviews have strict deadlines
By agreeing to review you take the responsibility of doing a thorough job
If you cannot commit to this, notify the editor asap
Editors understand you may not have the time, but are unforgiving if you commit and do a poor job
Good editors keep a list …
Class 4, 27/11/2006
FirstName LastName – Activity / Group

15. The right attitude: I can learn something!
Title
Humbleness and an open mind needed; 100% self-confidence can be harmful
Early assumptions on the correctness of the paper or the sufficiency of its references should be avoided
an elegantly written paper may have zero actual contribution
a paper with broken English may contain a major new idea
The papers recommended for acceptance should have novelty and be correct
If the reviewer can’t check a fact or is unsure, this should be stated in the review report
But don’t waste your time on analysing in detail a paper that is never publishable
a single crucial error is enough
Class 4, 27/11/2006
FirstName LastName – Activity / Group

16. Reviewing a research paper
Title
The paper to be reviewed is typically accompanied with a review form
fill the five point scale questions last
it is most important to write an itemized review report
Relevance
[ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent
Originality
Background knowledge of the subject and references
Technical content
Presentation
Class 4, 27/11/2006
FirstName LastName – Activity / Group

17. Reviewing a research paper: analysis
Title
The analysis of a paper can be done by generating explanations to the following eight points (Smith 1990)
What is the purpose of the paper?
Is the problem clearly stated and have the key issues been pointed out?
Is it clear what has been accomplished?
Is the paper appropriate for the intended forum?
If it is not, what could be a better choice?
Is the goal significant = has the work been worth doing?
Are the results just trivial variations or extensions of previous results?
Are there any new ideas, or novelties in research methodology?
Citation analysis using electronic libraries are a big help!
Class 4, 27/11/2006
FirstName LastName – Activity / Group

18. Reviewing a research paper: analysis (cont’d)
Title
Is the method of approach clear and valid?
Is there something fundamentally flawed in the approach?
Are the assumptions realistic and does that matter?
Is the method new? Can it be generalized to other problems?
Is the actual execution of the research correct?
Are the mathematics and statistics correct? Check!
Have the simulations been described in sufficient detail for replication?
What about the boundary conditions?
Do the results make sense?
This part may require considerable effort from the reviewer...
Class 4, 27/11/2006
FirstName LastName – Activity / Group

19. Reviewing a research paper: analysis (cont’d)
Title
Are the conclusions correct?
What are the applications or implications of the results and are the results analysed to an adequate depth?
Is the presentation satisfactory?
Is the paper readable? Is it structured according to the conventions of scientific publications?
What did you as the reviewer learn?
If you didn’t learn anything, then the paper is not publishable (provided that you understood the paper)
Class 4, 27/11/2006
FirstName LastName – Activity / Group

20. Reviewing a research paper: analyzing the references
Title
It is researcher’s professional obligation to cite prior work
the manuscript being reviewed includes claims of novelties; regularly citing prior research
the reviewer needs to check the validity of the claims
most efficient to carry out the analysis using electronic libraries
At minimum:
Check what is found using the key words of the article
Study the references you don’t know beforehand
Check which recent papers cite the same references
Check the references of those recent papers
Class 4, 27/11/2006
FirstName LastName – Activity / Group

21. Review structure The actual refereeing form
Title
The actual refereeing form
General comments on the paper
Specific comments on the paper
Confidential note to editor
General idea: be professional and non-hostile: write the review in a style that you would like to receive for your paper
Class 4, 27/11/2006
FirstName LastName – Activity / Group

22. The refereeing form
Title
Forms might look quite different but basically ask the same things
Poorly designed ones just have yes/no answers, good ones prompt the referee to elaborate
Make sure you read and understand it well
Class 4, 27/11/2006
FirstName LastName – Activity / Group

23. Writing the referee report
Title
No fixed rules exist, the following ones are according to (Smith 1990)
Most important: make your opinions clear; avoid ”perhaps” and ”maybe”; evaluate the paper, not the author; itemize the contributions
State the recommendation and its justification; the five point scale part of the evaluation form is not enough
Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers
Evaluate the significance and validity of the research goal
Evaluate the quality of methodology, techniques, accuracy and presentation; recommendations for revisions can be written here
Make a clear recommendation for or against publication with justifications
Class 4, 27/11/2006
FirstName LastName – Activity / Group

24. Compiling the recommendations
Title
Classification of papers (Smith 1990)
Very significant; includes major results (<1% of all papers)
Interesting work, a good contribution (<10%)
Minor positive contribution (10-30%)
Elegant and technically correct, but useless
Neither elegant nor useful, but not wrong
Wrong and misleading
Unreadable, impossible to evaluate
The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4)
Class 4, 27/11/2006
FirstName LastName – Activity / Group

25. Outcome Usually: You can only suggest, the choice is not yours
Title
Usually:
Accept the paper as it is
Paper requires minor changes
Paper requires major changes (with or without a new refereeing process)
Reject publication of the paper
You can only suggest, the choice is not yours
Decision is based on at least 3 reviews
Class 4, 27/11/2006
FirstName LastName – Activity / Group

26. Research proposals
Title
A research proposal is a request for funding submitted to,
MCyT, MECD, GENCAT
European Commission
NIH, NASA, NSF, ESF
other funding organization such as a foundation
The key difference to reviewing research papers is that
the reviewers also evaluate the proposers
Not all organizations use peer review as a means for selecting proposals for funding
Class 4, 27/11/2006
FirstName LastName – Activity / Group

27. Evaluating research proposals
Title
The evaluation criteria vary between funding organizations
Key criteria:
Is the research topic significant?
Are the goals realistic?
Has the proposer sufficient expertice and facilities to reach the goals?
Is the requested funding reasonable?
Class 4, 27/11/2006
FirstName LastName – Activity / Group

28. Ethics of refereeing Objectivity Fairness Speed
Title
Objectivity
Judge paper on its own merits
Remove prejudice
If you are not able to review it, return it
Fairness
Author may have different point of view / methodology / arguments
Judge from their school of thought not yours
Speed
Be fast, but do not rush. Author deserves a fair hearing
Class 4, 27/11/2006
FirstName LastName – Activity / Group

29. Ethics of refereeing Professional treatment Confidentiality
Title
Professional treatment
Act in the best interest of the author and conference/journal
Specific rather than vague criticism
Confidentiality
Cannot circulate paper
Cannot use without permission
Conflict of interest
Discuss with editor
Class 4, 27/11/2006
FirstName LastName – Activity / Group

30. Ethics of refereeing Honesty Courtesy
Title
Honesty
About your expertise and confidence in appraisal
Courtesy
Constructive criticism
Non-inflammatory language
Suggest improvements
Class 4, 27/11/2006
FirstName LastName – Activity / Group

31. Acting as an editor or program chairperson
Title
The editor
maintains correspondence with authors and referees
finds new referees if the ones assigned fail to act in given time
decides on acceptance, rejection or a revision round based on 2-4 review statements.
should distribute all review statements to the referees
receives occasional negative feedback
Review is not a vote! The editor is likely to line himself according to
the best justified recommendations
Conference program committees often rely on the numerical
evaluations, occasionally resorting to vote
Class 4, 27/11/2006
FirstName LastName – Activity / Group

32. How to become a referee
Title
Writing a publication that is cited is the most certain way to become a referee
Coordination or technical coordination of an EU RTD project is a direct road to proposal evaluations
Refereeing is very rewarding, helps to keep up-to-date and aware of developments in fields adjacent to ones own specialty
Class 4, 27/11/2006
FirstName LastName – Activity / Group

33. Final words Good referee reports are valuable and free of charge
Title
Good referee reports are valuable and free of charge
help in improving the paper
help in improving as a researcher
help in improving as a referee
Refereeing is a learning experience
Scientific progress rests heavily on peer reviews
Class 4, 27/11/2006
FirstName LastName – Activity / Group

34. PROJECT THESIS EDITOR OFFICE
Using the articles prepared, we will set up an editorial office
Each student will act as an associate editor of one article and will review three articles
Each student will peer review his/her three allocated articles and will return the referee report to the assigned associate editor
The associate editor will compile the final report and will return the final report with the individual referee reports to the Editorial Office
Title
Class 4, 27/11/2006
FirstName LastName – Activity / Group

35. Proposal Writing Class 4, 27/11/2006 Title
FirstName LastName – Activity / Group

36. Proposal Writing
Title
In order to carry out research, in general financing is required.
There are several national and international sources of funding and the process for obtaining funding is realised through proposal submission and review.
The aim of this section is to INFORM you of the proposal process, proposal formats and existing funding bodies.
The homework of this class will be to draft a proposal of your PhD to assist you in your resaerch planning, but NOT with a view to preparing a formal proposal in the style of those submitted for financing!
Class 4, 27/11/2006
FirstName LastName – Activity / Group

37. Content Why research ? Why should this be in a competitive context ?
Title
Why research ?
Why should this be in a competitive context ?
Why a research proposal ?
Getting started
What makes a good proposal ?
Writing your proposal
How to structure your proposal ?
The review process
Allocation of funding
What next ?
Getting help with your proposal ?
Quick TIPS for writing a good proposal
Class 4, 27/11/2006
FirstName LastName – Activity / Group

38. Why research ?
Title
Why is the development of research within universities a must ?
To maintain the quality of teaching programs.
Provide the basis for undergraduate and graduate thesis research projects.
Universities should be more than degree delivering institutions.
Universities should be the basket for new knowledge and developments.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

39. Why should this be in a competitive context ?
Title
Do universities have the financial capacity to develop and support research activities ?
Where can the money be found to develop and support research ?
How can society get the highest return on investment ?
Class 4, 27/11/2006
FirstName LastName – Activity / Group

40. Why a research proposal ?
Title
Convince others the project you have designed is important, worth the effort.
Convince others that you have the ability to carry out the research design and report the findings.
Generate funds to sustain the research units operation.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

41. Getting started
Title
Know your subject. The reviewers will look for an up-to-date knowledge of the research area.
Know your funder. Be aware of the priorities and interests of the funder you approach, and know that funders are unlikely to support the same idea twice.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

42. Getting started Consult colleagues.
Title
Consult colleagues.
Don’t be afraid to discuss your proposal with colleagues, or even with the grants officer at the funding body.
Early discussions can ensure that your proposal is targeted appropriately.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

43. What makes a good proposal ?
Title
A well-prepared application should require minimal effort on the part of the reviewer.
Proposals must demonstrate high scientific quality.
The requested funds must be in proportion to the proposed project (cost-effectiveness).
Class 4, 27/11/2006
FirstName LastName – Activity / Group

44. Writing your proposal
Title
Allow plenty of time to prepare your proposal. A good starting point is to write a one-page summary of the whole project. This may take a while to get right, but once completed it will serve as an invaluable tool for writing your full proposal.
Use your proposal to show the need and then fill the gap.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

45. Writing your proposal
Title
Present your proposal in terms of the aims and objectives of the funder and not just your own – make it clear how you will be helping them to fund their priorities.
Consider the questions the funder will be asking: Why fund you ? Why fund this ? Why now ? ... and make sure that the proposal answers them!
Class 4, 27/11/2006
FirstName LastName – Activity / Group

46. Writing your proposal
Title
Be aware that you will have limited to none opportunities to answer queries arising from a reading of your proposal.
Consult the funders website and read clearly the call for research proposals as well as the criteria against which your proposal will be judged.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

47. Writing your proposal
Title
Although it is the content that matters, good presentation is often crucial to making your proposal accessible to reviewers and keeping their interest.
Use diagrams and tables to add clarity;
Bullet points and sections can break up text;
Keep to page, word and font size restrictions; and
Activate the spell checker while writing.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

48. How to structure your proposal ?
Title
Check guidelines carefully – failing to meet the funder’s format and specifications is one of the most common reasons for applications being returned.
A common proposal structure normally consists of: title, abstract, background, aims and objectives, methodology, work program, resources, outcomes (outputs & dissemination), project management, reviewers.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

49. How to structure your proposal ?
Title
Title: This is the first impression the reader gets.
The title should be short and clear, and the reviewer should be able to understand from the title the intentions of the research.
A catchy title posing a question or including an apparant contradiction or acronym may be more easily remembered by a reviewer.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

50. How to structure your proposal ?
Title
Abstract: Should be a concise summary of the WHOLE project.
Use the abstract to identify the need for this research, state what you intend to do, and how you intend to do it.
Do not include unnecessary detail; make each phrase count.
And remember it is the first impression a reviewer gets of an applicant’s worth!
Class 4, 27/11/2006
FirstName LastName – Activity / Group

51. How to structure your proposal ?
Title
Background: This section should be used to put the work into context: what has been done before, and how will the proposed work add to it ?
What is the innovative aspect in the research project ?
Build your case by demonstrating your capability and familiarity in the area.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

52. How to structure your proposal ?
Title
Aims and objectives: The aims should describe what you intend to achieve by doing this piece of work.
Your objectives are the small steps you need to reach in order to achieve your aim.
Aims and objectives should be realistic, consistent, and link them to methods, timetable, and outcomes.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

53. How to structure your proposal ?
Title
Methodology: Methods should be detailed and well thought through.
Explain why you have chosen a particular method.
Base your explanation on literature references.
If your own experience of a methodology is limited, consider working with collaborators.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

54. How to structure your proposal ?
Title
Work program: Make use of a Pert chart to illustrate the building blocks – work packages – of the research project. Be detailed in the description of the content of each work package (why, objectives, method(s), duration, when are you going to carry out each WP, partners involved in the realization, sequence of WP, etc.).
Class 4, 27/11/2006
FirstName LastName – Activity / Group

55. IRRIGATION SCHEME LEVEL
Example of a Pert chart
Title
WP 1
WP 3
WP 2
WATERSHED LEVEL
Determination topography of reservoir
Determination of sediment load
Determination of rainfall and runoff
Life expectancy reservoir
INPUT for decision making
Modeling process
Gross water availability
Topography
Land use
Soil
Climate
IRRIGATION SCHEME LEVEL
Management system (CERES)
Water rights
WP 5
WP 4
Predicting the values of the irrigation and economic indicators for alternative scenarios of water management
Irrigation indicators
Irrigation infrastructure
Conveyance efficiency
WP 6
Calibration and validation of the methodology for the actual water management situation
WP 7
Economic indicators
WP 8
Simplified management system
Water demand estimation
Class 4, 27/11/2006
FirstName LastName – Activity / Group

56. How to structure your proposal ?
Title
Work program: This section contains also a diagrammatic work plan, called a Gannt chart.
The Gannt chart or diagrammatic work plan should also be accompanied by a written description.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

57. Example of a Gannt chart (= diagrammatic work plan)
Title
deliverables
Class 4, 27/11/2006
FirstName LastName – Activity / Group

58. How to structure your proposal ?
Title
Resources: The proposal should contain a detailed budget.
The budget asked should be in proportion to the volume and complexity of the work activities.
Be aware that funders vary as to what they are prepared to pay in terms of direct project costs, such as staff and equipment, and indirect costs, such as overheads.
The funder might request to approve beforehand own inputs or inputs from other institutions participating in the project.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

59. How to structure your proposal ?
Title
Outcomes, outputs (+ deliverables) and dissemination: In this section one should describe the contribution to knowledge and importance for future research, the benefits to users, and the broader relevance to beneficiaries.
Highlight how results will be disseminated (publications, conferences, commercial exploitation, websites, ....).
Class 4, 27/11/2006
FirstName LastName – Activity / Group

60. How to structure your proposal ?
Title
Project management: This might not be required for small projects.
However for projects in which several partners are involved sufficient information has to be provided on how the project will be managed (timescales, milestones, communication, criteria to measure progress, how crisis situations and conflicts will be handled, etc.).
Class 4, 27/11/2006
FirstName LastName – Activity / Group

61. Example of Project Organization chart
Title
Class 4, 27/11/2006
FirstName LastName – Activity / Group

62. How to structure your proposal ?
Title
Reviewers: Often requested to suggest name of referees.
Choose people who know you and your work;
Don’t use reviewers within your own institution;
Use international reviewers; and
Be aware that applicant’s own referees can write unfavourable reports.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

63. The review process
Title
Expert assessment: Traditionally applications will be assessed by 2 to 3 reviewers selected from the pool of experts.
Reviewers will make an independent assessment of the scientific quality of the proposal.
To be selected for funding at least 2 of the 3 reviewers should provide a positive assessment.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

64. The review process What are reviewers looking for ?
Title
What are reviewers looking for ?
High scientific quality;
Proposals that meet the funder’s priorities or fill a knowledge gap;
Novelty and timeliness;
Value for money;
A clear and well thought out approach; and
An interesting idea – catch their attention!
Class 4, 27/11/2006
FirstName LastName – Activity / Group

65. The review process
Title
Awards committee: Ranks the submitted proposals on the basis of the reviewer’s reports. Their operation and procedures can be very variable from funder to funder. They might for policy reasons of the funder deviate from the reviewer’s assessment.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

66. Allocation of funding
Title
Position in the ranking is important – it could mean the difference between success and failure. Proposals are often ranked into the following categories:
Fund;
Fundable;
Invite resubmission (used by some funders); or
Reject.
Class 4, 27/11/2006
FirstName LastName – Activity / Group

67. What next ? If the project is retained for funding  OK.
Title
If the project is retained for funding  OK.
If the project is found fundable  ???
If invited for resubmission  revise proposal  feedback from the reviewers panel.
If rejected, can be very frustrating  do not give up, try to get feedback  remember it is a learning process !
Class 4, 27/11/2006
FirstName LastName – Activity / Group

68. Quick for writing a good proposal
Title
Allow plenty of time;
Start by writing a summary of your proposed project;
Demonstrate an up-to-date knowledge of your field;
Present your proposal in terms of the aims and objectives of the funder;
Avoid jargon – say what you mean in clear, simple language;
Don’t be afraid to state the obvious;
Use graphics and diagrams to summarise what you are trying to communicate
Class 4, 27/11/2006
FirstName LastName – Activity / Group

69. Quick for writing a good proposal
Title
Allow a maximum of 4 charts (PERT, GANNT, PROJECT ORGANIZATION and BUDGET) - but include as many schematic representations of the concepts as possible;
Anticipate questions that may arise, before they arise;
Ask a colleague to review your proposal; and
Be enthusiastic about your idea – if you don’t sound interested, why should anyone else be ?
Class 4, 27/11/2006
FirstName LastName – Activity / Group

70. Funding Sources EUROPE
Title
EUROPE
European Comission ( - 4 year programmes with identified priorities and objectives. Currently Framework 6 - Framework 7 soon begins this year
European Science Foundation (
National Funding ( US
National Institute of Health (
DARPA (
NASA
Department Of Energy
Department Of Agriculture
Class 4, 27/11/2006
FirstName LastName – Activity / Group

71. Homework Draft a proposal of your PhD project
Title
Draft a proposal of your PhD project
The proposal should include:
Title Page
Table of Contents
Overall and sub-objectives
State-of-the-art and novelty of project
Workplan - divide into ‘workpackages’, for each WP describe the tasks and sub-tasks (a couple of paragraphs for each), the resources required, risk analysis and contingency plan, as well as deliverables and milestones
Pert Chart
Gantt Chart
Bibliography
(? Budget)
Class 4, 27/11/2006
FirstName LastName – Activity / Group