1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 PRAG Procurement Service Contracts – version November 2010 Advance module – Day II RCP-CIC By: Berk Babila, Managing Director - Ecorys in Turkey Ankara, 16 March 2011

2 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Content Terms of Reference −Purpose and role in the procurement process −Structure and templates −Objectives formulation −Results and assumptions −Estimation of inputs and costs −Requirements of personnel −Reporting and Evaluation

3 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference -Purpose and role in the procurement process -Structure and templates -Objectives formulation -Results and assumptions&risks -Estimation of inputs and costs -Requirements of personnel -Reporting and Evaluation

4 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Purpose and role of the ToR in the procurement process A key document, which describes the terms and conditions of a task to be performed A document that sets out the working relationship between the parties and communicates the mode for undertaking the assignment A document points out the responsibilities of the parties and the expected deliverables of the assignment

5 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference What is ToR? ToRs provide clearly detailed parameters for: why and for whom the operation is being done what is intended to be accomplished how it will be accomplished who will be involved and when milestones will be reached and project will be completed

6 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Objectives of a ToR To ensure that: ambiguities during the preparation of proposals, contract negotiation and execution of services are cleared out the specific task is carried out as outlined in the OIS the Consultant, undertaking the task, will focus on the major issues and the most serious, likely impacts of the task the task is undertaken within the time-frame planned and the budget allocated risks of unnecessary extra work, delays and additional expenses are reduced

7 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference ToR and other key documents ToR has to be in compliance with OIS -Objectives -Assumptions -Pre-conditions -Risks ToR has to be in compliance with FA - commencement date and period of execution ToR has to be in compliance with PRAG – template ToR has to be in compliance with State Aid Provisions ToR becomes Contract!!!

8 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Responsibilities for ToR The SRER submits ToR and budget estimate to the OS OS revises ToR in accordance with checklists and submits to CFCU/EUD EUD gives ex-ante approval after verification During the implementation it is SRER’s responsibility to closely monitor the performance of the Consultant’s experts and assure a qualitative and timely implementation of the project along with ToR At the end of the operation a final assessment of the Consultant’s performance will need to be carried out to check compliance with ToR OS/CFCU is in charge for contractual issues incl. payments and approval of expenses with ToR

9 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Types Depending on specificity of contract Technical assistance (fee-based) -to provide the experts specified in the contract -to manage or supervise a project Studies (global) -studies for identification and preparation of projects -feasibility studies, economic and market studies -technical studies -evaluations and audits Mixed Depending on size of contract/procedure

10 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Types (2) Depending on size of contract/procedure ≥ € International Restricted Tender Procedure > € but < € Framework Contracts, Competitive Negotiated Procedure ≤ € Single Tender

11 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference −Purpose and role in the procurement process −Structure and templates −Objectives formulation −Results and assumptions&risks −Estimation of inputs and costs −Requirements of personnel −Reporting and Evaluation

12 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Structure There are no universal ToRs, which will be suitable for any assignment However, there is a general structure, which should be observed when preparing ToRs ToR should start with a brief description and includes objectives, activities, results, experts profile and reporting A general structure of ToR has been drawn up by the EC in accordance with the principles of project cycle management The aim is to ensure that all issues are covered systematically and that key factors related to clarity of objectives and sustainability are thoroughly examined

13 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Structure of ToR for TA 1.Background information 2.Objective, purpose and expected results 3.Assumptions and risks 4.Scope of the work 5.Logistics and timing 6.Requirements 7.Reports 8.Monitoring and Evaluation

14 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Purpose and role in the procurement process Structure and templates Objectives formulation Results and assumptions&risks Estimation of inputs and costs Requirements of personnel Reporting and Evaluation

15 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Objectives Clear definition of the purpose of the proposed task and expected output for the assignment It should be kept up-to-date during implementation -results and activities may change as the project progresses -Project objective and purpose should not change

16 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Formulation objectives Before preparing the TOR the purpose must be clear The project purpose plus the assumptions in the same row of the logframe matrix should lead to the overall objective What outputs are required must also be clear! Remember – purpose should be achieved by the end of the project Overall objective – why the project is important Specific objective – why the project is needed Check with the operation identification sheet!!!

17 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Purpose The single central objective of the project in terms of sustainable benefits to be delivered to the project's beneficiaries Derived from the core problem that the project is trying to address One project – one purpose -If there is more than one purpose, it is likely that the purposes are repetitive, or that they overlap with results, or that the project is trying to address too many problems at once and should be split

18 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Purpose (2) The near-term impact and the immediate result of producing outputs By definition, project purpose is outside of the direct control of the project implementation team The team can be held accountable for producing a set of deliverables (outputs), but ultimately there is a set of important external factors outside of their control (assumptions) which may prevent them from accomplishing the Purpose Multiple purposes tend to weaken design and diffuse the efforts of implementation teams

19 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Distinguish between Purpose and activities Purpose and results - If the results lead to the project purpose without any assumptions in the results row of the logframe, then either the purpose is not ambitious enough, or conversely, the results are too ambitious. -For example, the result "National network of testing laboratories and certification and inspection bodies functioning" is practically indistinguishable from the project purpose "Technical and management capabilities of quality infrastructure bodies developed". The purpose should be formulated along the lines of "Free movement of goods acquis enforced in relevant sectors".

20 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Check Overall objectives -Do the overall objectives explain why the project is important for society / sectoral development? -Are the overall objectives inline with those elaborated in the operation identification sheet? Specific objective -Is there only one specific objective? -Is it in line with the project fiche? -Does the specific objective express a direct benefit for the target group? -Are SMART indicators identified? -Do these enable measuring the achievement of the specific objective?

21 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Service contract for TA Overall objective Purpose -For larger/complex projects there can be more than one purpose (i.e one per project component)

22 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Remember Objectives of the Consultant's assignment is: To carry out certain services under a Program/Component/Project The contractual objectives of assigning a Consultant will often be narrower and should therefore be clearly formulated in the Terms of Reference as e.g.: −advice and support (technical assistance) to the implementing agency −management of a component or sub-component, −study, master plan, survey, investigation −detailed engineering design, preparation of tender documents and assistance in tendering and award −review (of a study, a design or an investigation) −contract administration and construction supervision, take-over and commissioning −training and upgrading of staff, etc.

23 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Exercise Objectives formulation Setting the overall objective and purpose

24 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference −Purpose and role in the procurement process −Structure and templates −Objectives formulation −Results and assumptions&risks −Estimation of inputs and costs −Requirements of personnel −Reporting and Evaluation

25 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Results Immediate benefits for the end recipients Often outputs are stated here to better define the workload of Consultant Expressed as a change Compliant with project purpose and contributing to overall objective

26 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Presentation of results Either -in order of importance or −in chronological order −in line with objectives and listed (numbered) by project purpose or by component −very detailed (under global price) −based on Logframe −İn line with OIS (relevant results) −be quantified, timed and verifiable (SMART) −relevant to the TA Contract Do not mix results with objectives! State results for which the Consultant could be held responsible!!!

27 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Assumptions and Risks External factors beyond the control of the Consultant but important for the implementation of the project -define the environment and sustainability issues -summarise factors which the project cannot, or chooses not to control -are factors which the project does not address, but should monitor Assumptions and risks should be aligned with the OIS Consultant should address risks in their methodology

28 Advanced Training PRAG Services March 2011 If results are delivered and assumptions hold true, then the project purpose will be achieved Pre-condi-tions MeansCost Objectively Verifiable Indicators Intervention Logic Sources of Verification Assumptions Overall Objectives Project Purpose Results Activities

29 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Assumptions and Risks (2) Does the TOR address the assumptions outlined in OIS? Is the probability of realisation of the assumptions acceptable? Are pre-conditions outlined in OIS fulfilled?

30 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Three questions for risk assessment Cost: does the risk have a cost impact? Schedule: does the risk have an impact on the time schedule (timing of outputs and results of the measure)? Output and results: does the risk have an impact on the outputs and results of the measure? If not, then we are not dealing with a real risk

31 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Exercise Results and assumptions Setting the assumptions and risks

32 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference −Purpose and role in the procurement process −Structure and templates −Objectives formulation −Results and assumptions&risks −Estimation of inputs and costs −Requirements of personnel −Reporting and Evaluation

33 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Division of costs under fee-based contracts 1.Fees: cost for hiring the expert plus administrative expenditures for executing the contract and profit 2.Incidental expenditures: other direct cost for the contract (reimbursable) 3.Provision for expenditure verification This division is done to ease the financial management of the service contract (not to check tickets, telephone bills etc.) Administrative costs are real costs for the consultants and they will charge you for it

34 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference What is covered by fees? Cost for experts: Cost for hiring the staff (salary, fees, insurance, leave) Travel from the place of residence of the expert to the base of operation(s) Per diems / accommodation in the base of operation -If the base of operation is Ankara - any travel of key expert and non-key experts to Ankara is included within the fee rates -Per diems for Ankara are also included within the fee rates (for all experts) -If the experts travel from base of operation (Ankara) to other cities within Turkey or Europe for specific missions, the travel and per diem expenses for these missions are included within incidental expenditure

35 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference What is covered by fees? (2) Office-related costs: Office Rent Purchase or rental of office equipment Communications – fax, telecommunications, mail, courier etc. Consumables – not related to direct actions of Contract Secretarial services – for experts work – office managers, secretaries, interpreters, etc.

36 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Rules The Consultant is paid for days actually worked - 7 hours – working day for short-term experts (report input in hours) - 22 days a month for long-term experts (report in days) Time spent for travelling exclusively and necessarily for the purpose of the Contract may be included in the number of days or hours, as appropriate, recorded in the timesheets The Consultant's personnel is not be entitled for either sickness or casual leave provided (Project Manager may permit the Consultant to take unpaid leave during the period of execution of the contract when agreed with SRER)

37 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Incidental expenditure Cover non-fee direct cost related to the contract: –Travel / per diems within the target region outside base of operation –Publications –Direct costs related to training – premises, equipment, accommodation, interpretation, training materials, food or per diems for participants –Study tours, etc. –Translations of documents, etc. –Organisation of information campaigns –Setting up of a database –Development of MIS, etc.

38 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Expenditure verification Relates to the fees of the auditor who has been charged with the expenditure verification of the contract A budget heading with the introduction of detailed expenditure verification report A very detailed report – sufficient budget should be allocated (1-1.5% of contract value)

39 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Estimation of costs Necessary to estimate the value of the contract Includes following stages: –Estimation of human resource input –Estimation of other inputs –Valuation

40 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Human resources Estimate type of expertise that you need for the execution of tasks (senior/junior, and by function – training expert, procurement and contracting expert etc.) Estimate number of days for each task set in the ToR –Divide input on senior/junior Estimate total number of days (senior/ junior) Add inputs for project management – Team leader (if full time – input equals project implementation period) Estimate the fees budget – you may use Framework Contracts costs for valuation

41 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Travel and per diems under incidentals Include only travel costs and subsistence allowances for missions to be undertaken from the base of operations in the beneficiary country: -Travel – economy class -Per diem – EU maximum rates Rough estimate is needed Note that some contracts may require intensive travelling – monitoring of projects, surveys, etc.

42 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Logistics and timing Project office: –How many office(s) and office space (10 sqm for each expert ) –Provided by the consultant / or beneficiary Training premises (provided by the consultant / or beneficiary) Check the logic of requested inputs Specify the project start/end date Time of implementation

43 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Support staff Support staff is paid by the Consultant: –Project director –Quality manager –Project assistant(s) –Secretary(ies) –Interpreter(s)/Translator(s) –Driver(s) Put requirements on backstopping staff only if really needed for the implementation (in exceptional cases)

44 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Other inputs Publications – translation, interpretation etc. Direct costs for studies Training – total number of training days (man/days of training = number of participants x days of training) Study tour: total number of man/days of study tours

45 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Incidental expenditure Estimate total value of incidental expenditures: –Add your calculations for publications, training, etc. –Add an error margin not less than 10% The provision for incidental expenditure should be stated in ToR as a lump sum. Do not give breakdown of costs! (otherwise the amounts identified become legally binding) Provision for incidental expenditure in ToR is an upper limit on the incidental costs. Only actual costs will be paid to the consultant.

46 Advanced Training PRAG Services March 2011 Value of contract DaysFee rateValue Fees Team leader Senior experts Junior experts Incidentals Expenditure verification 1% Total plus error

47 Advanced Training PRAG Services March 2011 From consultant’s view Days/ months Fee rateValue Senior experts + TL Per diems/ accommodation Tickets Junior experts Office rent Secretarial support Equipment Consumables/ communication Incidentals Expenditure verification 1% Management fee 5-8% Total actual cost

48 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Exercise Estimation of inputs and costs Work on estimations for HR and incidentals

49 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference –Purpose and role in the procurement process –Structure and templates –Objectives formulation –Results and assumptions&risks –Estimation of inputs and costs –Requirements of personnel –Reporting and Evaluation

50 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Type of experts Key experts Other experts Support staff and backstopping

51 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Type of experts (2) Team Leader Senior Experts Junior Experts FWC Category I – 15 years Category II – 10 years Category III – 5 years Short term – experts working less than 6 consecutive months of the contract Long term – experts working more than 6 consecutive months of the contract

52 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Key experts All experts who have a crucial role in implementing the contract are referred to as key experts. –Key expert 1: Team Leader –Key expert 1: Training Expert Recommended maximum of 4 key experts - otherwise the maximum scores will be negligible and evaluation will be difficult.

53 Advanced Training PRAG Services March 2011 Terms of Reference The division of scores between O&M and Key experts, as well as the maximum scores for each category must be adapted according to the requirements of the specific tender procedure. The no. of key experts must correspond to the number of key expert profiles identified in the Terms of reference and should not exceed 4 key experts. Evaluation grid

54 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Key expert requirements Identify: –Qualifications and skills: professional technical skills, team management skills, communication skills, and/or language skills as appropriate –General professional experience: as appropriate –Specific professional experience: as appropriate Differentiate between minimum requirement and preferred requirement, if applicable, for GPE and SPE Be sure such experts exist in the market! Do not give the profile of the ‘ideal expert’

55 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Key expert requirements (2) The criteria should be as broad as possible. "preferably 10 years experience…but a minimum of 5 years required” "a university degree in Economics or equivalent professional experience” (number of years should be stated) in order not to automatically disqualify offers with experts who have 40 years of relevant experience but which lacks the formal university degree if you put “university degree in Economics" and the tenderer proposed an expert with degree in Law … the offer should be rejected as deviating from the technical award criteria

56 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Key expert requirements (3) Profiles should be non-discriminatory Participation in tendering procedures shall be open on equal terms to all eligible persons No discriminatory requirements for sex, age, nationality The ToRs may not require an expert defined as local expert (defined as an expert with permanent residence in the concerned country). The nationality or place of residence may not be considered as a criterion. This is not legal and can not be indicated in the request as it is discriminatory You may use profiles developed by other ToR – EU/ World bank/ etc., but do not forget to adapt them

57 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Other experts CVs for experts other than the key experts are not examined prior to the signature of the contract They should not be included in tenders During the technical evaluation, the only aspect is considered for the non-key experts is whether the number of working days are sufficient for the requirements of the ToR to be achieved In their proposal tenderer’s must indicate whether they are to be regarded as long-term/short-term and senior/junior in order to be clear which fee rate in the budget breakdown will apply to each profile

58 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Other experts (2) The selection of short-term experts after the contract start is subject of approval by the CA The selection procedure used by the Consultant to select these other experts must be transparent, and based on pre-defined criteria, including professional qualifications, language skills and work experience. Selection must be in line with Organisation and Methodology Note that civil servants and other staff of the public administration of the beneficiary country cannot be recruited as experts. Prior EUD Approval!!!

59 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Experts’ input It is recommended that the precise time inputs of the experts be left to the discretion of tenderers. Identify only absolute minima for the contribution of key experts. The minimum % of time which experts should work in the beneficiary country could be specified, e.g 75%, to limit the possibility that experts spend a large proportion of their time outside the beneficiary country to which they are supposedly giving technical advice performance of the contract (and therefore payment) is based on working days only

60 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Experts’ input The days and hours of work of the Consultant or the Consultant’s personnel in the country of the CA shall be fixed on the basis of the laws, regulations and customs of the country of CA and the requirements of the services Consultant's personnel shall not be entitled to either sick or casual leave provided Fee rates for all experts must include all the "administrative costs of employing the relevant experts, such as relocation and repatriation expenses Max 220 days a year (60 days paid leave by the Project Manager) The schedule of estimated numbers of working days provides a direct basis for calculating the maximum contract value

61 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Exercise Requirements of personnel Draft a job description, and requirements for the Team Leader; in particular Qualifications&Skills, General Professional Experience, Specific Professional Experience

62 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference –Purpose and role in the procurement process –Structure and templates –Objectives formulation –Results and assumptions&risks –Estimation of inputs and costs –Requirements of personnel –Reporting and Evaluation

63 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Reports In this section only progress reports (on activities) has to be listed (reports additional to the ones stated in the Scope of work) Types of reports -Inception report -Interim reports must be prepared every six months during the period of execution of the contract. -Final report Do not ask for more reports if you have other ways to follow the performance of the consultant: -The consultant charges you for this reports -You waste time for reading and approving the reports

64 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Do not forget! Monthly progress reports!!! MMM!!! Reports for Steering Committee meetings!!! All these reports are additional to any required in Section 4.2 of Terms of Reference

65 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Contents Each report consist of a narrative section and a financial section. The financial section contains the details: - time inputs of the experts -incidental expenditure -expenditure verification; -Invoice Do not ask for more reports if you have other ways to follow the performance of the consultant: -The consultant charges you for this reports -You waste time for reading and approving the reports

66 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Submission and approval Specify copies of the progress reports Specify language of reports –By contract these are in EN –If you want translation – specify –Specify hard/electronic Specify bodies responsible for the approving the progress reports -sequencing Specify to whom copies should be send

67 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 Terms of Reference Monitoring and Evaluation 8.1. Definition of indicators: –Specific performance measures chosen because they provide valid, useful, practical and comparable measures of progress towards achieving expected results. –Can be quantitative: measures of quantity, including statistical statements; or qualitative: judgements and perception derived from subjective analysis 8.2. Special requirements: If appropriate

68 This project is co-financed by the European Union and the Republic of Turkey Advanced Training PRAG Services March 2011 End of Day II Closing Questions and Answers