Measuring Usability of Biometrics Review of Experiences at NPL Linda Johnstone Sorensen

2 Outline 10 Years ago –BIOTEST project views on Usability Testing for Biometrics 1997–2008 –Observations on usability measurement during … NPL Performance & Usability Testing 1999 NPL Performance Evaluations 2000 – 2005 UKPS Biometrics Enrolment Trial 2008 –How would we update proposals from BIOTEST? What works well ? What doesnt? What else should be included?

3 BIOTEST: Collaborative EU Project Objective: Develop methodologies to measure biometrics systems performance Focussed on –Accuracy (I.e. error rates such as FMR, FNMR) –Security (I.e. robustness to spoofing, etc.) –Usability (Measuring ease of use etc.) 10+ years later … –Methodologies & metrics for assessing usability remain the least well established

4 BIOTEST (1997) Usability measure proposals What to measure? –Ease of enrolment & use –Acceptability of enrolment & use –Invasiveness of enrolment & use –Levels of Supervision –Enrolment risks –Exceptional enrolees Not defined as usability –Physical characteristics Dimensions of device Interfaces Environmental conditions Context of use How to measure? –Quantitative measures Effectiveness –E.g. successful enrolment Efficiency –E.g. proportion of unproductive time –Time taken vs that of experienced user, … –Qualitative measures Expert assessment Subject/Operator feedback

5 Usability testing of biometric systems for ATMs Evaluation conducted in 1998/99 –Assessment of verification accuracy and usability –to guide a consideration of implementing biometrics in an ATM system –2 fingerprint & 2 face recognition systems –Opportunity to apply some of the methodology developed in BIOTEST

6 Usability testing of biometric systems for ATMs –Verification accuracy assessed by Scenario Tests Emulating sample of typical bank customers: 200+ test subjects from staff on Teddington site and some relatives Emulating enrolling bank clerk: NPL staff on project team –In-depth usability assessment 20 subjects (demographic balance) Enrolment, training, & verification Separate observer Videotaping of the interactions Subjects interviewed before and after trial –Open questions Short questionnaire for all 200 test subjects –Closed questions

7 Change in opinions during the trial OpinionsBefore using devicesAfter the trial Not comfortable using biometrics Face: 4 / 20 Fingerprint: 4 / 20 Face: 6 / 20 Fingerprint: 3 / 20 Preferred biometric Face: 4 / 20 Fingerprint: 4 / 20 No preference: 12 / 20 Face: 5 / 20 Fingerprint: 9 / 20 No preference: 6 / 20 Other perceptionsFingerprints perceived as easy to forge Fingerprints viewed as more stable than faces Confidence in system improved by rejections (when doing something wrong)

8 Comparison with other methods of verification Users asked how biometric devices compared to using a PIN –8/20 reported that biometric devices felt safer than PIN –9/20 positive to biometrics not requiring memorisation –10/20 said they would be willing to use the biometric system to take out cash at an ATM –The reservations expressed loss of confidence due to problems experienced during the trials and hence

9 Enrolment and verification problems observed Face recognition –Height –Problems with eyes (e.g. infections) –Wearing items such as glasses, hats, sunglasses –Variations in hairstyle –Time taken to enrol/verify Fingerprints –Poor quality fingerprints (e.g. due to manual labour or accidents) –Finger placement (e.g. just the tip of finger on sensor) –Removing finger before image capture is complete

10 NPL 2005 evaluation Impact of usability for operators NPL Biometric Evaluation 2005 –Pier 2-3 Handheld Iris camera –Holding the camera steady Expected to be difficult Found easy after a practicing –Intrusiveness Expected that subject would find the experience intrusive (camera held close to face). Findings – Operators also feel uncomfortable holding the camera so close –Performance differences between operators not significant

11 NPL 2005 evaluation Impact of usability for operators Correct Thicker reference lines on forehead and chin NPL Biometric Evaluation 2005: 3D-face enrolment –Of the errors incurred, most attributable to poor enrolment –Operator Instructs subject throughout Raises/lowers camera Checks subject –Positioning –Pose –No smiling/talking –Fringe … –Problems not always clear on on-screen display Incorrect Too farToo close (d (due to height)

12 NPL 2005 evaluation Impact of usability for operators NPL Biometric Evaluation 2005: 2D-face enrolment –Warnings & operator advice for non-optimal images –Resulting in … Perfect matching performance –No false matches and –No false non-matches Longer enrolment times Shows multiple quality measures & whether these are adequate Recommendation to accept or retake Shows whether algorithm correctly locates eyes

13 UKPS biometrics enrolment trial Feasibility study on biometric ID cards –Main performance unknowns are around usability by all sections of the population 2004/5 Biometrics enrolment trial –Focus on Enrolment & verification durations Customer perceptions and reactions Exception cases Demographic differences –Included a significant proportion of disabled users demographically balanced quota group volunteering members of the public

14 Error rate by demographic group 1 st attempt enrolment errors FaceIrisFingerprint Age: %21%27% Age: %18%29% Age: %17%27% Age: %21%29% Age: %26%30% Age: %30%31% Age: 65+4%41%34% Hearing impairment10%51%35% Learning disability12%56%63% Physical impairment12%52%50% Visual impairment12%65%36% Outlier groups show more usability problems

15 Problem with measuring participant experiences and perceptions What drives feelings of positivity towards having facial, iris and fingerprint biometrics recorded? Positivity measured as: –Level of concern about the technique after demonstration –Favourability towards its adoption See biometrics as strengthening of the security of ones passport Preventing identity fraud Preventing illegal immigration/working –Combined measure of a participants experience and attitude towards the biometric devices

16 Context of use seems more influential than experience of use Drivers for positivity analysed as: –Time taken for biometric capture –Level of intrusion –Ease of positioning –Level of initial concern More than twice as influential as ease of positioning and time taken

17 UKPS Biometrics Enrolment Trial Actual time taken vs. User feedback on time taken –User response not particularly correlated with actual time taken

18 Findings on quantitative usability measures Effectiveness –Measured by error rates Failure to Enrol, False Non-match rate, False Match Rate –These are and mainly determined by exception cases Exception cases (oldest / youngest / tallest / shortest / disabilities) often reveal more usability issues than typical users Efficiency –Appropriate to measure use with Habituated subjects (familiar with using the systems) as well as Unhabituated subjects, with & without operator assistance Many tests use mainly unhabituated subjects with assistance

19 Findings on qualitative usability measures Expert Assessment –Useful part of many evaluations: revealing errors due to usability –Assessor independent of the operator/subject interaction –Some (though not all) issues can be assessed by experts without observing real use Possibility of checklist? Subject/Operator feedback –Limited usefulness as indicator of usability Users goals not always the same as the systems goals –Feedback is influenced by factors other than operational effectiveness E.g. Pre-conceptions Novelty of the experience

20 Outstanding issues - 1 Acceptability of a biometric system Classes of users –People problems; some groups of people have problems with due to physical appearances –Technology may be challenging for certain user groups –Ergonomic concerns for specific user groups User behaviour –Lack of behaviour compliance; when people dont do what you want them to do (ideal system do not require explanations). Operator concerns with biometric systems

21 Outstanding issues - 2 How do we combine measures of usability in biometric systems? –We cannot always see what is most usable from measures of performance alone –We cannot see what is most usable from user feedback alone –Most errors observed in trials with biometric systems are not errors of the system, but errors in its use! Trust –How do we include trust? Trust in the technology –E.g. people like to see a failure once in a while, to confirm that the system works Trust in the system –E.g. security of the system –Storage of biometric data – remotely or in personal chip-card

22 Thank you!