SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking

We revisit the memory update mechanism in SAM2-based visual object tracking and identify confidence-only mask selection as the dominant cause of drift under occlusion, rapid motion, and distractors. We introduce SENTRY, a training-free, plug-and-play, refine-before-write module that validates each memory update for short-horizon temporal consistency before committing it. SENTRY aggregates diverse segmentation hypotheses per frame and backtracks each candidate to form short, temporally coherent tracklets. A neighbor-aware, cycle-consistent matching stage evaluates candidates against recent trajectories to favor temporally and geometrically consistent hypotheses. SENTRY requires no retraining and leaves the base architecture untouched; it simply replaces confidence-driven writes with consistency-validated ones. To ensure fair and comprehensive evaluation, we re-evaluate major open-source SAM2-based trackers across all available scales and datasets, filling gaps where prior works reported incomplete results. Integrated into five strong baselines, SENTRY delivers consistent gains across nine benchmarks, achieving new zero-shot SOTA on LaSOT, LaSOText, GOT-10k, VOT20, VOT22, and DiDi. It also yields strong improvements elsewhere, even when all baselines are re-evaluated across all model scales. Despite its checks, SENTRY remains real-time, running at 32.8 FPS, introducing approximately 25% overhead relative to SAM2. Our results provide the first unified, all-scale evaluation of SAM2-based trackers and demonstrate that enforcing temporal validity at write time is a general design principle that consistently stabilizes memory-augmented tracking without retraining.