System vs Algorithm Evaluations in Face Presentation Attack Detection

In the field of face presentation attack detection (face-PAD) or liveness detection, evaluating the effectiveness of detection systems and algorithms is crucial to ensure robust and reliable anti-spoofing measures. Two primary evaluation approaches used in PAD research are system evaluation and algorithm evaluation. In this blog post, we will explore the key differences between them, their objectives, and how they contribute to advancing the state-of-the-art in face presentation attack detection.

System evaluations determine the effectiveness of an anti-spoofing solution

System evaluation in face presentation attack detection involves assessing the overall performance of an anti-spoofing system or solution. It focuses on evaluating the system as a whole, considering factors such as camera capture and user interaction, preprocessing techniques, and decision fusion methods. Its primary objective is to determine the effectiveness and reliability of the complete anti-spoofing solution in real-world scenarios. System evaluation typically involves the following steps:

• Presentation Attack Instruments (PAI) selection and creation. Choosing and creating a diverse set of PAI that simulates various spoofing techniques found in real-world scenarios.
• System performance metrics. Adopting recognized performance metrics, such as those defined in ISO 30107, to measure the effectiveness of the system, including false acceptance rate (FAR), false rejection rate (FRR), and other relevant metrics specific to the PAD problem, such as Bona fide Presentation Classification Error Rate (BPCER), Attack Presentation Classification Error Rate (APCER) or Average Classification Error Rate (ACER).
• Systematic experiment design. Designing experiments to study the system’s performance under various conditions, including different attack types, varying environmental factors, and different data acquisition setups in a way that covers as much ground as possible in order to simulate real world attacks.

An example of system evaluation is the one proposed by the Spanish Ministry of Defense through its National Cryptologic Centre (CCN-CPSTIC) for qualifying video identification systems (in Spanish). Alice Biometrics has been qualified as a multiplatform high security video identification service.

Algorithm evaluation gain insights into the strengths and limitations of algorithms and their applicability to attack scenarios

Algorithm evaluation in face presentation attack detection focuses on assessing the performance of individual algorithms or methods (frequently based on AI approaches) used for anti-spoofing. It aims to analyze the effectiveness of specific algorithms or models in detecting and differentiating between genuine and spoofed faces. The objective of algorithm evaluation is to gain insights into the strengths and limitations of different algorithms and their applicability to various attack scenarios. Algorithm evaluation typically involves the following aspects:

• Benchmark dataset. Collecting and curating a benchmark dataset that is representative of real-world scenarios, containing diverse presentation attacks and variations.
• Algorithmic performance metrics. Defining appropriate performance metrics tailored to algorithmic evaluation, such as detection accuracy, precision, recall, and specific metrics relevant to the PAD domain.
• Comparative analysis and reporting. Conducting comparative analysis of algorithms, ensuring fair and reproducible comparsions, and reporting evaluation results transparently.

An example of algorithm evaluation is the one proposed by the NIST FRVT PAD. Alice Biometrics has participated in the first PAD assessment conducted by NIST ranking among the top PAD providers worldwide.

Understanding the lights and shadows of algorithm evaluation and system evaluation

Algorithm evaluation offers several benefits, including:

1. Fair comparison. By evaluating PAD algorithms on standardized datasets, this method enables fair comparisons between different vendors or research groups. This allows an objective assessment of the performance of various techniques and facilitates advancements in the field.
2. Focused analysis. It allows researchers to deeply analyze the strengths and weaknesses of specific PAD algorithms. Which, at the same time, can lead to incremental improvements in detection accuracy and the development of more robust techniques.

However, algorithm evaluation also has limitations, such as:

1. Limited representation. Evaluating algorithms on specific datasets may not fully capture the diversity of real-world presentation attacks, potentially limiting the generalizability of the results. The datasets used in algorithm evaluation might not encompass all possible spoofing scenarios and may not adequately represent emerging attack techniques.
2. Lack of system-level insights. Algorithm evaluation does not account for the performance and integration challenges that arise when deploying the algorithm within a complete face recognition system. Consequently, the evaluation results obtained through algorithmic evaluation might not fully reflect the performance of the algorithm in real-world operational settings.

System evaluation, on the other hand, offers several advantages, including:

1. Real-world relevance. It assesses the performance of the entire face recognition system, taking into account various factors like environmental conditions and attack scenarios, providing insights into real-world deployment challenges. 
2. Usability considerations. System evaluation considers factors like user experience, processing time, and integration compatibility, ensuring the system’s practicality and ease of use.

However, system evaluation also has some drawbacks, such as:

1. Complexity and variability. Evaluating the entire face recognition system involves considering multiple components and their interactions. Variations in hardware, software, environmental conditions, and attack scenarios make it challenging to reproduce the exact conditions under which the evaluation was conducted. This frequently leads to difficulties in fair comparison between systems.
2. Limited system or platform scope. Since system evaluation needs to be performed end-to-end with individuals carrying out attacks and entering to the system, even if the process follows standard recommendations (such as ISO 30107), the scope of the evaluation is limited to the physical implementation of the system or device and, therefore, cannot be reproduced in a different environment.

Advancing Face Presentation Attack Detection

Both system and algorithm tests play vital roles in the evolution of face presentation attack detection techniques. System evaluation provides a holistic view of the overall anti-spoofing solution, allowing researchers and developers to identify areas of improvement, optimize system configurations, and validate their effectiveness in real-world scenarios. It considers factors like environmental conditions and usability, ensuring the system’s practicality. However, system evaluation faces challenges in terms of reproducibility and fair comparison between systems due to the complexity of the evaluation process.

On the other hand, algorithm tests focus on the detailed analysis of individual techniques, which allows researchers to develop more accurate and robust algorithms for detecting increasingly sophisticated presentation attacks. They also enable fair comparisons between different vendors or research groups by testing specific PAD algorithms on standardized datasets. It enables objective assessments and incremental improvements in detection accuracy. But algorithm tests have restrictions regarding limited representation and the lack of system-level insights.

By conducting system and algorithm tests regularly, the face presentation attack detection community is able to drive innovation, enhance the state-of-the-art, and develop effective anti-spoofing solutions that can withstand evolving spoofing techniques. Collaboration between researchers, data providers, and industry stakeholders is vital to ensure standardized evaluation protocols, diverse benchmark datasets, and transparent reporting of evaluation results.