1. Russ Ryan, Vice President National Biometric Security Project The Importance of Biometric Testing

2.  Biometrics for National Security (BiNS)  National Signatures Project  National Energy Technology Lab (NETL)  NIST  International Organization for Migration (IOM)  Office of Presidential Affairs (UAE)  International Labour Organization (ILO)  BioAPI Consortium  State of West Virginia NBSP National Biometric Security Project National Biometric Security Project

3. Biometric Applications Biometric Applications HSPD-24 GAO ?  Robust biometric passports  Financial and medical services authorizations  Border and travel services  Drivers’ licenses  Physical and Logical access

4.  Increasing reliance on biometrics to secure access, transactions & identity  Equally increasing demand for accurate, unbiased evaluations  Testing can provide accurate metrics on how the technology will perform in the real world  Alleviating unfounded concerns about operational performance Understanding Biometric Performance Understanding Biometric Performance

5.  Universal  Unique  Permanence  Collectable  Performance  Acceptance  Spoof Resistance Attributes of an Ideal Biometric Attributes of an Ideal Biometric

6. Biometric Testing Today Biometric Testing Today  Performance of biometric systems is a function of:  strength of the underlying biometric.  quality and information content of the input  configuration and architecture of the system  the relationship of accuracy and throughput  error rates, the nature of failures and their cost, and system vulnerabilities which contribute to an overall assessment of system performance  Increasingly, biometric devices are components of larger systems imposing external variables that impact biometric system performance in the field

7. Biometric Testing Today Biometric Testing Today  Three major considerations in testing biometric products  dependence of measured error rates on the application  need for a large test population  necessity for a time delay between enrollment and testing

8. Comparison of Testing Types Comparison of Testing Types  Technology Testing  Goal: Produce a repeatable and scalable assessment of an algorithm/sensor using offline data processing  Scenario Testing  Goal: Determine overall system performance (both algorithmic & human factors performance measures)  Operational Testing  Goal: Determine biometric system performance in a specific environment with a specific target population * Best Practices in Testing and Reporting Performance of Biometric Devices, by A. J. Mansfield, National Physical Laboratory and J. L. Wayman, San Jose State University. Published 2002 by The Centre for Mathematics and Scientific Computing,National Physical Laboratory, Queens Road, 88, Middlesex, England.

9. Technology Testing Technology Testing  Understand/compare software techniques used to acquire, process and compare biometric data  Main focus is on the pattern matching technique used to compare biometric data  Evaluates different classification and matching methods on efficiency, speed and performance  Offline processing of data carried out in laboratory  Evaluation compares competing algorithms  from a single type of technology  carried out on a standardized database  collected by a universal sensor  results determine the relative effectiveness of the tested algorithms

10. Scenario Testing Scenario Testing  Evaluates performance across biometric devices  Each system has its own acquisition sensor and receives different data inputs than those tested in technology (algorithm) evaluation  Data collected for all tested systems must come from same environment and same population  Test results are only considered repeatable under identical control variables & environment Scenario evaluation helps an end user decide which biometric device has the potential to work best for his/her needs

11. Operational Testing Operational Testing  Determine performance of a biometric system in a real application environment  Population and environment are not controlled  System vulnerability can also be performed Helps determine how system as a whole will perform by testing a live system in its native environment for its intended application

12. Conformance Testing Conformance Testing  Determines conformance with relevant published ISO/IEC standards  Utilizes conformance test suites designed for specific standards  Evaluations will expand to include additional standards as the software modules are written and field tested Standards Evaluated Target Value INCITS 377-2004Pass/Fail INCITS 378-2004Pass/Fail ISO 19794-2-2005Pass/Fail INCITS 379-2004Pass/Fail INCITS 381-2004Pass/Fail INCITS 385-2004Pass/Fail ISO 19794-2-2005Pass/Fail INCITS 396-2005Pass/Fail INCITS 395-2005Pass/Fail ILO SIDPass/Fail ICAO LDS 1.7Pass/Fail BioAPIPass/Fail

13. Vulnerability Testing Vulnerability Testing  Impersonation attempts (disguises) or spoofing (artifact substitution for live feature)  Database attacks (exchanging or corrupting references)  Tampering with threshold settings  Network-based attacks Product “vulnerabilities” must be defined in the context of the operating environment and proper usage within the design parameters of the product

14. Interoperability Testing Interoperability Testing  Multi-modal systems demand acceleration of biometric interoperability  Interoperability testing assesses  ability to exchange and use information on a single system in a multi-modal environment  interface of the biometric component with the holistic security program

15. Interoperability Trade-offs… Interoperability Trade-offs… Lowers complexity of the application - Re-use - Future Proofing - Vendor independence - Upgrade path - Simplifies CM - Simplified integration -Product optimization - Better performance -Lower level control - More sophistication -Can be faster to market (due to standards development time) May incur additional overhead - May not be able to take advantage of vendor unique capabilities - Interfaces are generic and consensus based, so may not be optimized for a particular use -Custom interfaces for each proprietary product to be interfaced - Increased cost/complexity - Added CM - Product changes affect application - Can result in vendor dependence Standard Proprietary Advantages Disadvantages Courtesy of Cathy Tilton, VP Standards & Technology, Daon

16. Usability Testing Usability Testing  Intuitiveness of the system interface with the user community  Is the transaction an inviting and positive experience?  Is consistent instruction and feedback built into the process?  Is the performance reliable for operational staff as well as users?

17. Qualified Product List Testing Qualified Product List Testing  First initiated and commercialized by NBSP  Utilizes comprehensive scenario test capability  Initially used to identify products that successfully passed common performance thresholds  Increasingly tailored to the application

18. QPL Testing Benefits QPL Testing Benefits  Catalog of commercially available products that meets minimum standards for a specific application  Significant reduction in duplicative pilot tests  Acceleration of acquisition process by identifying a field of suitable products  Opportunity for vendors’ to demonstrate general or specified performance capabilities

19. Factors Affecting Biometric Performance Factors Affecting Biometric Performance  Variations in:  biometric pattern  the way users present the biometric  the way the sensor reads the biometric  System scalability  the transmission process (including noise introduced by compression & expansion)  User acceptance/application- specific limitations

20. Additional Measurement Parameters Additional Measurement Parameters  Reliability, availability, scalability, maintainability  Security, including vulnerability to spoofing  Human factors, including user acceptance  Cost/benefit in comparison to existing security processes and systems  Privacy regulation compliance

21. Laboratory Certification Laboratory Certification  BSI awarded ISO/IEC 17025 Accreditation  specifies requirements for competency to conduct biometric tests  covers testing performed using standard methods, non- standard methods and laboratory-developed method  laboratory customers, regulatory authorities and accreditation bodies use it to confirm the competency of laboratories.  NIST  NIST Handbook 150-25 with technical requirements and guidance for accreditation of laboratories under the NVLAP Biometrics Testing program released Sept. 2009

22. Russ Ryan, rryan@nationalbiometric.orgrryan@nationalbiometric.org 703-201-8179 www.nationalbiometric.org The Importance of Biometric Testing