1. Lecture 7 Evaluation

2. Purpose Assessment of the result Against requirements Qualitative Quantitative User trials Etc Assessment of and Reflection on process Actions

3. Purpose of Evaluation To demonstrate that requirements have been met An assessment of the fitness for purpose To identify shortcomings To identify opportunities To reflect upon product/process etc Demonstrate you did it right – and to learn

4. Purpose The aim is to honestly assess the results and processes to Demonstrate successful completion of the project Improve the product or identify potential improvements Learn from mistakes

5. Methodology for Evaluation There are many different ways to evaluate systems/products etc There may be software or methodological tools to support some forms of evaluation E.g. see web page assessment tools Note: Evaluation is NOT testing

6. Methodology for Evaluations There are many dimensions to the evaluation: Subjective v Objective Qualitative v Quantitative Analytical v Experimental System v Component Perspectives: Customer v User v Developer …. …….

7. Evaluation against Requirements Requirements should define a set of criteria that the product must meet. So, check whether they are met Are they sufficiently precise? Are they measurable? How can they be assessed? Develop a methodology for evaluation – before development

8. Evaluation against Requirements Aside: Acceptance tests A precise set of tests that must be passed for acceptance Necessary but not sufficient Defined in contract Some overlap with evaluation Aside:  -testing,  -testing …

9. Subjective v Objective assessment For some criteria there will be hard, measurable features For others, we have to rely upon an assessors judgement

10. Qualitative v Quantitative Many features can be assessed quantitatively Others may have to be qualitative Excellent, good, acceptable, poor Ranking

11. Quantitative assessment What can we measure? File size Speed To perform a task Latency Memory requirements Reliability etc

12. Qualitative measures Attractiveness Ease of use Fitness for purpose Extensibility etc Often these are the more important criteria

13. Perspectives Who are the stakeholders? Customer User Developer Maintainer Etc Look at each perspective

14. What is being evaluated? Some assessments may focus upon different parts of the artifact The whole system The user interface Documentation The implementation Maintainability Reliability Extensibility etc

15. Trials An evaluation from the users’ perspective may require trialling with typical users Assess user’s reaction to the work How? May have different classes of user Expert, novice Frequent, occasional

16. How to evaluate Develop a plan for the evaluation Identify the criteria Specify how they will be assessed Plan will necessarily focus on some aspects – at the expense of others

17. Reporting Define the evaluation methodology Report the results of the evaluation Discuss recommendations Improvements Changes Enhancements

18. Reflection Evaluate the process What was good What was bad Recommendations How to improve it next time

19. Actions Review the set of recommendations Product Process Identify which of these Must be addressed within the project Can be addressed within the project Should be carried forward to future work