OMEN

We introduce class-incremental motion forecasting, a novel setting in which new object classes are sequentially introduced over time and future object trajectories are estimated directly from camera images, and cOntinual Motion PrEdictioN (OMEN), the first approach specifically designed to address this novel setting.

Our approach is the first to tackle the problem of class-incremental motion forecasting. In contrast to (a) traditional and (b) end-to-end motion forecasting, (c) the underlying model is trained incrementally, with access to labels only for a subset of all classes $C^n$ and raw multi-view camera images. As a result, it continually learns to forecast the motion of all classes in an end-to-end manner, while handling imperfect object detections and successfully combating catastrophic forgetting.

Abstract

Motion forecasting enables autonomous vehicles to anticipate scene evolution by predicting the future trajectories of dynamic agents. However, existing approaches typically assume a closed-world setting with a fixed object taxonomy and access to high-quality perception, limiting their applicability in the real world where perception is imperfect, and new object classes may emerge over time. In this work, we introduce class-incremental motion forecasting, a novel setting in which new object classes are sequentially introduced over time and future object trajectories are predicted directly from camera images. We propose the first end-to-end framework for this setting, which adapts to newly introduced classes while mitigating catastrophic forgetting of previously learned ones. Our method generates motion forecasting pseudo-labels for known classes and matches them with 2D instance masks from an open-vocabulary segmentation model. This 3D-to-2D keypoint voting mechanism filters inconsistent and overconfident predictions, while a query feature variancebased replay strategy samples informative past sequences to preserve prior knowledge. Extensive evaluations on nuScenes and Argoverse 2 show that our approach successfully preserves performance on known classes while effectively adapting to novel ones. We further demonstrate zero-shot transfer to realworld driving and show that the framework extends naturally to open- and closed-loop end-to-end class-incremental planning on nuScenes and NeuroNCAP.

Technical Approach

Illustration of the proposed OMEN architecture. At each incremental step $i$, we create detection and motion forecasting pseudo-labels for the old categories with the old model $\Phi^{i-1}$, filter them in an offline process via a 3D-to-2D keypoint voting mechanism with the 2D instance masks output by a VLM, and add them to the detection ($\triangle$) and motion forecasting ($\circ$) ground truth of $D^i$. Furthermore, a replay buffer is created based on the latent motion query space of the old model.