Leveraging Semantic Segmentation for Image Manipulation Detection and Localization

Detecting and localizing image manipulation has long been a focus in computer vision. Current state-of-the-art methods primarily identify visual and JPEG compression artifacts. In this study, we propose the integration of semantic segmentation to enhance object awareness and improve the accuracy of spliced object localization. Instead of treating semantic segmentation as an independent submodule, we integrate it as a third branch of an end-to-end model. This approach balances visual, compression, and segmentation artifacts, reducing overemphasis on any single branch. Extensive evaluations on established datasets show a three percent average IoU increase in performance across five mainstream datasets for image manipulation localization. We also provide insights into how existing digital defense models adapt to new image tampering techniques like generative fill and expand.