Real-time human-computer interaction using eye gazes

Eye gaze emerges as a unique channel in human–computer interaction (HCI) that recognizes human intention based on gaze behavior and enables contactless access to control and operate software interfaces on computers. In this paper, we propose a real-time HCI system using eye gaze. First, we capture and track eyes using the Dlib 68-point landmark detector, and design an eye gaze recognition model to recognize four types of eye gazes. Then, we construct an instance segmentation model to recognize and segment tools and parts using the Mask Region-Based Convolutional Neural Network (R-CNN) method. After that, we design an HCI software interface by integrating and visualizing the proposed eye gaze recognition and instance segmentation models. The HCI system captures, tracks, and recognizes the eye gaze through a red–green–blue (RGB) webcam, and provides responses based on the detected eye gaze, including the tool and part segmentation, object selection and interface switching. Experimental results show that the proposed eye gaze recognition method achieves an accuracy of >99 % in a recommended distance between the eyes and the webcam, and the instance segmentation model achieves an accuracy of 99 %. The experimental results of the HCI system operation demonstrate the feasibility and robustness of the proposed real-time HCI system.