Feature extraction through local learning

RELIEF is considered one of the most successful algorithms for assessing the quality of features. It has been recently proved that RELIEF is an online learning algorithm that solves a convex optimization problem with a margin-based objective function. Starting from this mathematical interpretation, we propose a novel feature extraction algorithm, referred to as local feature extraction (LFE), as a natural generalization of RELIEF. LFE collects discriminant information through local learning and can be solved as an eigenvalue decomposition problem with a closed-form solution. A fast implementation of LFE is derived. Compared to principal component analysis, LFE also has a clear physical meaning and can be implemented easily with a comparable computational cost. Compared to other feature extraction algorithms, LFE has an explicit mechanism to remove irrelevant features. Experiments on synthetic and real-world data are presented. The results demonstrate the effectiveness of the proposed algorithm.