Error analysis of pattern recognition systems - The subsets bootstrap

A biometric system is an instance of a pattern recognition system with well-defined error conditions, which allows objective statistical error analysis. Biometrics is an emerging technology with fierce competition between many manufacturers using a variety of biometrics, such as, fingerprints, for recognizing human identities. Manufacturers of biometric systems are continuously refining the technology and claiming "high accuracy." This may be defined simply as a system that works or as a system that makes very few or no errors. Obviously, such loose definitions of accuracy are undesirable and there is a need for a precise definition. We argue that biometric match score accuracy is best expressed in terms of a curve, the Receiver Operating Characteristic curve. More importantly, we argue that confidence intervals, or margins of error, should be provided for this curve. This allows for determining whether accuracy differences between systems are really statistically significant. We introduce a novel bootstrap technique for computing the confidence regions of the error estimates and compare them to a commonly used parametric method. This bootstrap technique is inspired by the moving blocks bootstrap, which samples with replacement from blocks of data thereby accounting for dependence among the data. Our approach samples with replacement from specifically determined subsets of the data. We call this new bootstrap technique "the subsets bootstrap."