Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos

Abstract

We address the problem of online (streaming) action seg-mentation for egocentric procedural task videos. While pre-vious studies have mostly focused on offline action segmen-tation, where entire videos are available for both training and inference, the transition to online action segmentation is crucial for practical applications like AR/VR task assistants. Notably, applying an offline-trained model directly to online inference results in a significant performance drop due to the inconsistency between training and inference. We propose an online action segmentation framework by first modifying existing architectures to make them causal. Sec-ond, we develop a novel action progress prediction module to dynamically estimate the progress of ongoing actions and using them to refine the predictions of causal action segmen-tation. Third, we propose to learn task graphs from training videos and leverage them to obtain smooth and procedure-consistent segmentations. With the combination of progress and task graph with casual action segmentation, our frame-work effectively addresses prediction uncertainty and over-segmentation in online action segmentation and achieves significant improvement on three egocentric datasets.11Code is available at https://github.com/Yuhan-Shen/ProTAS.