CrowdDeep: Deep-learning from the crowd for nuclei segmentation

In recent years, deep convolutional neural networks (CNNs) have shown tremendous success in solving many biomedical tasks. However, the development of deep convolutional networks requires access to large quantities of high-quality annotated images for training and evaluation. As image annotation is a tedious task for biomedical experts, recruiting non-expert crowd workers is an economical and efficient way to provide a rich dataset of annotated images. We present an approach to improve the accuracy of segmenting nuclei in Hematoxylin and Eosin (H&E) slides by hiring crowd workers. We first present a crowdsourcing framework that enables fast and efficient acquisition of nuclei-segmented masks from the crowd by providing manual and semi-automatic annotation methods. Then, we present CrowdDeep, a novel technique to improve the accuracy of deep learning models trained on expert annotation by efficiently hiring crowd-annotated data. CrowdDeep consists of two sub-networks: Crowd-Subnet, and Expert-Subnet. The Crowd-Subnet is trained on the crowd-annotated images to extract crowd-related features from the crowd-annotated masks, while the Expert-Subnet is trained on the expert-driven annotations to extract expert-related features from the expert-annotated masks. Then, it calculates the final segmentation mask from the generated segmentation masks by two sub-networks. The results show that CrowdDeep outperforms a CNN model trained on solely expert-derived annotations in terms of F1-Score, IOU, and Pixel Accuracy. This approach is multi-organ and generalizes across different organs, staining, and disease states and is easily expandable by crowdsourcing images with an assortment of nuclei shapes and sizes from any desirable body tissue.