CA-Fuse-MIL: Cross-Attention Fusion of Handcrafted and Deep Features for Whole Slide Image Classification

Whole Slide Image (WSI) analysis plays a pivotal role in computer-aided diagnosis and disease prognosis in digital pathology. While the emergence of deep learning and self-supervised learning (SSL) techniques helps capture relevant information in WSIs, directly relying on deep features overlooks essential domain-specific information captured by traditional handcrafted features. To address this issue, we propose fusing handcrafted and deep features in the multiple instance learning (MIL) framework for WSI classification. Inspired by advancements in transformers, we propose a novel cross-attention fusion mechanism “CA-Fuse-MIL,” to learn complementary information from handcrafted and deep features. We demonstrate that Cross-Attention fusion outperforms WSI classification using either just handcrafted or deep features. On the TCGA Lung Cancer dataset, our proposed fusion technique boosts the accuracy by upto 5.21% and 1.56% over two different set of deep features baseline. We also explore a variant of CA-Fuse-MIL which utilizes multiple cross-attention layers.