UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model

Abstract

Traditional unsupervised optical flow methods are vul-nerable to occlusions and motion boundaries due to lack of object-level information. Therefore, we propose UnSAM-Flow, an unsupervised flow network that also leverages object information from the latest foundation model Segment Anything Model (SAM). We first include a self-supervised semantic augmentation module tailored to SAM masks. We also analyze the poor gradient landscapes of traditional smoothness losses and propose a new smoothness definition based on homography instead. A simple yet effective mask feature module has also been added to further ag-gregate features on the object level. With all these adaptations, our method produces clear optical flow estimation with sharp boundaries around objects, which outperforms state-of-the-art methods on both KITTI and Sintel datasets. Our method also generalizes well across domains and runs very efficiently.