Action anticipation predicts future actions from partially observed videos. Because observations are limited, it requires inferring hidden intentions, leading to two uncertainties: (1) observation-level uncertainty from partial or noisy visual evidence, and (2) intention-level uncertainty because the same observation can imply multiple plausible future intentions. Prior work largely models only observation-level uncertainty, limiting long-horizon anticipation where unobserved, evolving intentions drive behavior.

We introduce IntenDiff, a hierarchical diffusion framework that explicitly models intention-level uncertainty via sub-intentions: temporally evolving short-term intentions that bridge the video’s global intention and future actions. (1) Sub-Intention Diffusion starts from the global intention and stochastically samples sub-intention embeddings over time in a label-free manner. We ground these embeddings in language by co-fine-tuning with a vision-language model (VLM) using a contrastive alignment loss, producing multiple, linguistically coherent hypotheses beyond the visible frames. (2) Action Diffusion generates actions conditioned on sampled sub-intentions via uncertainty-aware conditioning, adaptively assigning higher weight to time segments where sub-intentions remain aligned with the global intention. By jointly capturing intention-level and observation-level uncertainty, IntenDiff yields intention-consistent and robust long-horizon predictions under partial observability. Empirically, IntenDiff achieves consistent gains across six benchmarks, advancing the state-of-the-art.