Pose invariant face recognition under arbitrary unknown lighting using spherical harmonics

We propose a new method for face recognition under arbitrary pose and illumination conditions, which requires only one training image per subject. Furthermore, no limitation on the pose and illumination conditions for the training image is necessary. Our method combines the strengths of Morphable models to capture the variability of 3D face shape and a spherical harmonic representation for the illumination. Morphable models are successful in 3D face reconstructions from one single image. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using spherical harmonics representation. In this paper, we show that we can recover the 3D faces with texture information from one single training image under arbitrary illumination conditions and perform robust pose and illumination invariant face recognition by using the recovered 3D faces. During training, given an image under arbitrary illumination, we first compute the shape parameters from a shape error estimated by the displacements of a set of feature points. Then we estimate the illumination coefficients and texture information using the spherical harmonics illumination representation. The reconstructed 3D models serve as generative models to render sets of basis images of each subject for different poses. During testing, we recognize the face for which there exists a weighted combination of basis images that is the closest to the test face image. We provide a series of experiments on approximately 5000 images from the CMU-PIE database. We achieve high recognition rates for images under a wide range of illumination conditions, including multiple sources of illumination.