TECIS: TEXT-ENHANCED CROSS-MODAL INSTRUMENT SEGMENTATION IN SURGICAL VIDEO

Surgical Instrument Segmentation (SIS) is crucial for enhancing surgical efficiency and safety but remains challenging due to visual similarities among instruments. Recent advancements, especially those leveraging the Segment Anything Model (SAM) due to its strong capability to generate accurate object masks, have improved segmentation accuracy yet still struggle with subtle semantic distinctions. We propose TeCIS (Text-enhanced Cross-modal Instrument Segmentation), a novel method integrating textual descriptions directly into the segmentation process. TeCIS empolys a cross-attention mechanism in the segmentation mask decoder, leveraging detailed instrument descriptions to convey semantic information that helps the model accurately identify and segment each surgical tool. Evaluations on EndoVis2017 and EndoVis2018 datasets show TeCIS substantially reduces semantic confusion and surpasses existing SAM-based methods, highlighting the effectiveness of integrating text for improved surgical segmentation.