Model-based integration of visual cues for hand tracking

We present a model based approach to the integration of multiple cues for tracking high degree of freedom articulated motions and apply it to the problem of hand tracking using a single camera sequence. Hand tracking is particularly challenging because of occlusions, shading variations, and the high dimensionality of the motion. The novelty of our approach is in the combination of multiple sources of information which come from edges, optical flow and shading information. In particular, we introduce in deformable model theory a generalized version of the gradient-based optical flow constraint, that includes shading flow i.e., the variation of the shading of the object as it rotates with respect to the light source. This constraint unifies the shading and the optical flow constraints (it simplifies to each one of them, when the other is not present). Our use of cue information from the entirety of the hand enables us to track its complex articulated motion in the presence of shading changes. Given the model-based formulation, we use shading when the optical flow constraint is violated due to significant shading changes in a region. We use a forward recursive dynamic model to track the motion in response to 3D data derived forces applied to the model. The hand is modeled as a base link (palm) with five linked chains (fingers) while the allowable motion of the fingers is controlled by recursive dynamic constraints. Model driving forces are generated from edges, optical flow and shading. The effectiveness of our approach is demonstrated with experiments on a number of different hand motions with shading changes, rotations and occlusions of significant parts of the hand.