S519: Evaluation of Information Systems Result D-Ch10

Putting them together Now we are ready to write our evaluation report. Basically we are going to fill our content to the checklist boxes we learned in lec2.

The Key Evaluation Checklist (D-p. 6-7) I. Executive Summary II. Preface III. Methodology 1. Background & Context 2. Descriptions & Definitions 3. Consumers 4. Resources 5. Values 6. Process Evaluation 7. Outcome Evaluation 8 & 9. Comparative Cost-Effectiveness 10. Exportability 11. Overall Significance 12. Recommendations & Explanations 13. Responsibilities 14. Reporting & Follow-up 15. Meta-evaluation

KEC Preliminary checkpoints To orient readers to the basics of : what the evaluation is and why the evaluation was done What the main approach was

I. Executive summary Shows a short overview (1-2 pages) of what was evaluated and what the main findings were. You can consider to include the followings: Very short description of the program The overall conclusion about the quality or value of the evaluation The graphical profile of the evaluand‘s performance (see Exhibit7.6) Several (<7) bullet points about the most important strengths and weaknesses.

II. Preface Address the following questions: Who asked for the evaluation and why? What are the main evaluation questions? Is this a formative or summative evaluation? Who are the main audiences for this evaluation report? Mainly based on your work in Ch2 Please always provide justifications for your conclusions

III. Methodology The main methodologies used in the evaluation How you set up your experiments How you collect data How you analyze your data? How you draw conclusion And each with why.

KEC foundations checkpoints To represent all the initial ingredients Background Context What it is Whom it serves The nature and limitation of any resources Where the values come from

1. Background and Context Incude just enough information for readers to be able to understand the basic retionale for the program Contents are: Why did this program or product come into existence in the first place (needs assessment) How is the evaluation supposed to address the needs and problems? What are the main context of evaluands (e.g. Physical, economic, political) to facilitate or constrain the evaluation?

2. Description and definitions Describe your evaluand in enough details Definition of important terms

3. Consumers Identify the actual and potential recipients or users of the evaluand Describe consumers in geographic location, demographics Downstream impactee

4. Resources Describe the available and important resources (which helps to understand the conclusion of the evaluation) Funds/budget Physical space Experience Networks

5. Values To convey to the audience how the evaluation team determinded what should be considered as „valuable“ or „high quality“ for the evaluand. Justify the validity

KEC Sub-evaluation checkpoints Getting evaluation explicitly

6. Process evaluation The issues of evaluation, content and implementation (see Ch4) Ethics, the priniciple of equity and fairness Consistency with professional and scientific standards Efficiency (i.e, minimal wasted effort or resource) Needs of consumers (e.g., timeliness, learning style, current knowledge) Needs of staff (e.g., performance on tasks and activities)

7. Outcome evaluation The outcomes are what happened to the consumers as a result of coming into contact with the evaluand Intended and unintended Short-term and long-term effects A good way to do List the main outcome dimensions with a brief explanation of how the list was generated. Rate each dimension on importance Explain the used methods (ch7) Rate the each outcome dimension (ch8) Provide the justification (evidence, logic, methods) for your rate

8&9. Comparative cost- effectiveness It is important to know whether the evaluand is cost-effective See D-p62-63

10. Exportability Whether your evaluation could also have value outside its current context. It is always worth considering the possibility

KEC conclusions checkpoints Combining Checkpoints6-10 to draw an overall conclusion

11. Overall significance Using synthesis methodology to draw a summary covering Checkpoints 6-10 Graphical representation (e.g., Exhibit7.6) Summary of strengths and weaknesses

12. Recommendations and explanations (possible) Recommendations for furture improvement

13. Responsibilities (possible) Who is repsonsible for good or bad results You have to be highly skilled

14. Report and support Who get the copies of the evaluation reports and in what forms What are follow-ups to ensure that the findings are used

15. Meta-evaluation Self review of the evaluation itself The validity of their conclusions Utility to relevant stakesholders The methodology used for evaluations Credibility Cost Lessoned learned during this evaluation.

Exercise Go back to your evaluation group Discuss all these checkpoints and how are you going to put all your result together

Exercise Go through these checkpoints Write down 3 checkpoints which are most important to you and state why? Discuss with your partner Form the group and select 3 most important checkpoints, why and Present your group conclusion and your suggestions in poster Hang your group poster and people can shop around poster Vote for the best poster