Segment and Track Anything

Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang +2 more
5/11/2023
cs.CV

Official code: z-x-yang/Segment-and-Track-Anything (3,138 stars, AGPL-3.0, last commit 2026-03-13).

Abstract

This report presents a framework called Segment And Track Anything (SAMTrack) that allows users to precisely and effectively segment and track any object in a video. Additionally, SAM-Track employs multimodal interaction methods that enable users to select multiple objects in videos for tracking, corresponding to their specific requirements. These interaction methods comprise click, stroke, and text, each possessing unique benefits and capable of being employed in combination. As a result, SAM-Track can be used across an array of fields, ranging from drone technology, autonomous driving, medical imaging, augmented reality, to biological analysis. SAM-Track amalgamates Segment Anything Model (SAM), an interactive key-frame segmentation model, with our proposed AOT-based tracking model (DeAOT), which secured 1st place in four tracks of the VOT 2022 challenge, to facilitate object tracking in video. In addition, SAM-Track incorporates Grounding-DINO, which enables the framework to support text-based interaction. We have demonstrated the remarkable capabilities of SAM-Track on DAVIS-2016 Val (92.0%), DAVIS-2017 Test (79.2%)and its practicability in diverse applications. The project page is available at: https://github.com/z-x-yang/Segment-and-Track-Anything.

View on arXivView PDF

Code Implementations(1)

z-x-yang/Segment-and-Track-AnythingOfficial100%
AGPL-3.0

An open-source project dedicated to tracking and segmenting any objects in videos, either automatically or interactively. The primary algorithms utilized include the Segment Anything Model (SAM) for key-frame segmentation and Associating Objects with Transformers (AOT) for efficient tracking and propagation purposes.

3,138355Shell, PythonApr 16, 20237 months agoAGPL-3.0
interactive-segmentationsegment-anythingsegment-anything-modelvideo-object-segmentationvisual-object-tracking

Similar Papers

Track Anything: Segment Anything Meets Videos

87% match1 repo7.0k2023

Tracking Anything in High Quality

85% match1 repo7522023

Training-Free Robust Interactive Video Object Segmentation

85% match2024

MITracker: Multi-View Integration for Visual Object Tracking

84% match1 repo1382025

Find First, Track Next: Decoupling Identification and Propagation in Referring Video Object Segmentation

84% match1 repo822025

SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation

84% match1 repo1822025

Cite this paper

@article{cheng2023segment,
  title  = {Segment and Track Anything},
  author = {Yangming Cheng and Liulei Li and Yuanyou Xu and Xiaodi Li and Zongxin Yang and Wenguan Wang and Yi Yang},
  year   = {2023},
  eprint = {2305.06558},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.06558v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.06558.svg)](https://codeofpaper.com/paper/2305.06558)

Discussion