1. S519: Evaluation of Information Systems
Meta-evaluation
D-Ch11

2. Meta-evaluation
It is evaluation of an evaluation (Scriven, 1991) to determin the quality and/or value of an evaluation

3. Five criteria
Validity
Utility
Conduct
Credibility
Costs

4. Validity
To what extent, the conclusions of the evaluation are justified
Check carefully the facts they use and values they assign
Question everything
Constraints (budget, time, etc.)
Using KEC checkpoints
Using standards (

5. Validity Covers all relevant sources of value
Comprehensively covers process, outcome and cost
Include no irrelevant or illicit criteria
Data used to directly address the criteria
Include analyses that are appropriate for the data
Clearly states how data are interpreted
Is clear about where evaluative conclusions come from
Include valid recommendation

6. Utility Are the findings
Relevant to the questions or decisions being faced by the audience
Timely
Clearly communicated
Cost effective

7. Conduct Legal Ethical Professional standards Cultrual appropriateness
Unobtrusiveness
Minimal disruptive to the evaluand

8. Credibility Familiar with the context
Independence, impartiality, and/or lack of conflict of interest
Expertise in evaluation and in the evaluation subject field

9. Cost
How reasonable the costs of an evaluation

10. Meta-evaluation rating table

11. Exercise D-p217, exercise 2 Form a pair, do the evaluation
Form a group to select the best result,
Design a poster
Present them in a poster
Using three sentenses to summarize your evaluation
What you learn
What you feel
How to improve