Butterfly-Net: Spatial-temporal architecture for medical image segmentation

Tetiana Klymenko, Seong Tae Kim, Kirsten Lauber, Christopher Kurz, Guillaume Landry, Nassir Navab, Shadi Albarqouni

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Citation (Scopus)


Radiation therapy tries to maximize the effect of radiation on the tumor and minimize its influence on adjacent tissues. However, it highly depends on the accuracy of the tumor segmentation on the planning radiography images. Tumor contouring today is carried out exclusively with the significant contribution of medical specialists which is a tedious, time-demanding, and expensive task. Further, it is prone to the inter-/intra-observer variation that can affect the reliability of the outcome. Existing methods for automatic tumor segmentation can reduce the influence of these factors, but are not completely reliable and leave a lot of room for improvement. In this work, we exploit Spatio-temporal information from the longitudinal CT scans to improve the deep neural network for tumor segmentation. For this purpose, we devise a novel volumetric Spatio-temporal memory network, Butterfly-Net, which stores the previous scan information and reads for the segmentation at the target time point. Moreover, the effect of clinical factors is investigated in the framework of our volumetric Spatio-temporal memory network. Experimental results on our longitudinal CT scans show that our model could effectively utilize temporal information and clinical factors for tumor segmentation. The code is made publicly available 1

Original languageEnglish
Title of host publication2021 IEEE 18th International Symposium on Biomedical Imaging, ISBI 2021
PublisherIEEE Computer Society
Number of pages5
ISBN (Electronic)9781665412469
Publication statusPublished - 13 Apr 2021
Event18th IEEE International Symposium on Biomedical Imaging, ISBI 2021 - Nice, France
Duration: 13 Apr 202116 Apr 2021

Publication series

NameProceedings - International Symposium on Biomedical Imaging
ISSN (Print)1945-7928
ISSN (Electronic)1945-8452


Conference18th IEEE International Symposium on Biomedical Imaging, ISBI 2021


  • Medical Volumetric segmentation
  • Spatio-temporal Memory Networks.


Dive into the research topics of 'Butterfly-Net: Spatial-temporal architecture for medical image segmentation'. Together they form a unique fingerprint.

Cite this