Lecture 7 Evaluation

Purpose Assessment of the result Against requirements Qualitative Quantitative User trials Etc Assessment of and Reflection on process Actions

Purpose of Evaluation To demonstrate that requirements have been met An assessment of the fitness for purpose To identify shortcomings To identify opportunities To reflect upon product/process etc Demonstrate you did it right – and to learn

Purpose The aim is to honestly assess the results and processes to Demonstrate successful completion of the project Improve the product or identify potential improvements Learn from mistakes

Methodology for Evaluation There are many different ways to evaluate systems/products etc There may be software or methodological tools to support some forms of evaluation E.g. see web page assessment tools Note: Evaluation is NOT testing

Methodology for Evaluations There are many dimensions to the evaluation: Subjective v Objective Qualitative v Quantitative Analytical v Experimental System v Component Perspectives: Customer v User v Developer …. …….

Evaluation against Requirements Requirements should define a set of criteria that the product must meet. So, check whether they are met Are they sufficiently precise? Are they measurable? How can they be assessed? Develop a methodology for evaluation – before development

Evaluation against Requirements Aside: Acceptance tests A precise set of tests that must be passed for acceptance Necessary but not sufficient Defined in contract Some overlap with evaluation Aside:  -testing,  -testing …

Subjective v Objective assessment For some criteria there will be hard, measurable features For others, we have to rely upon an assessors judgement

Qualitative v Quantitative Many features can be assessed quantitatively Others may have to be qualitative Excellent, good, acceptable, poor Ranking

Quantitative assessment What can we measure? File size Speed To perform a task Latency Memory requirements Reliability etc

Qualitative measures Attractiveness Ease of use Fitness for purpose Extensibility etc Often these are the more important criteria

Perspectives Who are the stakeholders? Customer User Developer Maintainer Etc Look at each perspective

What is being evaluated? Some assessments may focus upon different parts of the artifact The whole system The user interface Documentation The implementation Maintainability Reliability Extensibility etc

Trials An evaluation from the users’ perspective may require trialling with typical users Assess user’s reaction to the work How? May have different classes of user Expert, novice Frequent, occasional

How to evaluate Develop a plan for the evaluation Identify the criteria Specify how they will be assessed Plan will necessarily focus on some aspects – at the expense of others

Reporting Define the evaluation methodology Report the results of the evaluation Discuss recommendations Improvements Changes Enhancements

Reflection Evaluate the process What was good What was bad Recommendations How to improve it next time

Actions Review the set of recommendations Product Process Identify which of these Must be addressed within the project Can be addressed within the project Should be carried forward to future work