01
Find the representation gap
Actions should inherit the scene's motion, instead of learning an unrelated coordinate system.
Vision-Induced Tokens for Action Learning
VITAL: From Visual Motion
to Precise Action
- 1 Horizon Robotics
- 2 Shanghai Innovation Institute
- 3 Fudan University
- 4 National University of Singapore
- 5 Zhejiang University
VITAL turns visual motion into a language for action: a scene-grounded scaffold paired with a normalized metric residual.
One flow evaluation predicts both components. A bounded residual update refines local precision while preserving the visual plan.
The intuition · in motion
A path. A gap. An idea.
A short visual walkthrough of how VITAL connects visual motion to precise action. Plays step by step when in view.
01 / 06 · MotivationConcept animation · not measured trajectories
Visual scaffoldDemonstrated trajectoryMetric correctionDecoded plan
Visual dynamics→Action targets?→Driving decisions
01 · Start with visual motion
Can actions speak the language of vision?
World-action models anticipate visual dynamics, yet their action targets are often defined independently. What if visual motion itself could define the action representation?
The question: connect visual grounding and metric precision.
Explore the motivation →Steps 2–4 explain target construction with recorded images and a demonstrated trajectory. Steps 5–6 use only deployment context. Geometric offsets are exaggerated for visibility.
The idea, step by step
Start with the gap.
End with a correction.
Scroll through the chain of observations that shaped VITAL. The large panel follows your place in the story.
01
The question
What does vision already capture?
World-action models predict useful motion, but their action coordinates are often chosen independently of visual dynamics.
02
Measure what vision leaves behind
The visual scaffold is smooth and meaningful, yet centimeter-scale errors still matter near road boundaries.
03
Split motion from metric precision
Two paired token groups give small corrections their own learning scale.
04
Make one evaluation executable
One action evaluation, one trajectory: 90.8 PDMS at 149 ms, including image encoding and residual correction.
05
Refine without redrawing the world
One-step before/after examples show local corrections to progress, road compliance, and TTC while preserving the predicted visual scaffold.
The method
A shared language for prediction and refinement.
Future images supervise the tokenizer during training. At deployment, VITAL uses only the current front-camera image, ego state, and navigation command.
1 · Visual scaffold. A visual encoder turns recorded image motion into four motion tokens and a decoder maps them to a trajectory.
2 · Paired residual. The remaining trajectory error is centered, projected into a PCA basis, and scaled so small metric corrections are learnable.
3 · Joint action learning. A flow-matching action expert predicts both token groups from deployment context, then a single Euler update decodes the plan.
4 · Residual refinement. A bounded actor changes only residual coordinates, leaving the predicted visual scaffold fixed.
40 points · 10 Hz4 s horizonNFE = 1one generated trajectory
01 · observeCurrent context
Front image + ego state + navigation command.
02 · predictJoint code
Visual tokens and residual tokens from one flow evaluation.
03 · decodeExecutable motion
Frozen decoder returns the four-second pose sequence.
04 · refineBounded correction
One residual update improves local execution.
Codec in motion
Watch the residual close the gap.
Each pair uses the same RGB sequence and fixed metric axes. The first clip shows the visual scaffold; the second adds its paired residual.
Pair 1 · scaffold
Visual motion, before correction
Clean-code reconstruction leaves 0.241 m ADE in this example. The maneuver is visible; local metric detail is not yet closed.
Pair 1 · scaffold + residual
Same scene, near the demonstrated path
The paired residual brings this example to 0.523 mm ADE. The full validation diagnostic is 0.311 mm ADE.
Pair 2 · scaffold
A second turning scene
The same scale-consistent gap appears again: 0.104 m ADE before the residual.
Pair 2 · scaffold + residual
Metric detail restored
The paired residual reduces this example to 0.904 mm ADE.
Evidence
Nearly all of the score, one action evaluation.
All policies use one generated trajectory. The v2 results are a no-retraining transfer of the v1-trained policy.
90.8 PDMS
NAVSIM v1 · navtest · NFE 1
Final VITAL policy.
10 steps: 91.0 PDMS
91.1 EPDMS
NAVSIM v2 · navtest · NFE 1
Corrected scoring, no retraining.
10 steps: 91.2 EPDMS
38.6 EPDMS
NAVSIM v2 · navhard · NFE 1
Reactive two-stage continuation.
10 steps: 38.1 EPDMS
| Protocol | NFE / plans | Score |
|---|---|---|
| NAVSIM v1 navtest · final policy | 1 / 1 | 90.8 PDMS |
| NAVSIM v1 navtest · final policy | 10 / 1 | 91.0 PDMS |
| NAVSIM v2 navtest · no retraining | 1 / 1 | 91.1 EPDMS |
| NAVSIM v2 navhard · no retraining | 1 / 1 | 38.6 EPDMS |
Why the split matters
Scaffold for the maneuver. Residual for the detail.
With recorded images and a ground-truth paired residual, ADE falls from 123.6 to 0.3 mm and FDE from 221.6 to 0.5 mm, as rounded in the current paper figure. Reconstruction PDMS rises from 92.1 to 94.7.
The mean x/y high-frequency velocity-energy fraction is 0.1% for the visual scaffold and 61.4% for the residual. This describes their measured behavior: the residual encodes whatever error remains, rather than a prescribed frequency band.
Codec diagnostic: recorded future images and the residual encoded from the demonstrated trajectory are supplied. These reconstruction numbers are distinct from deployed planning performance.
Current paper · Visual reconstruction and residual precision
Current paper · Planning quality, latency, and action learning
Action learning
Visual grounding helps before refinement.
Panel (c) compares action targets after one training pass, before residual RL: VITAL reaches 70.1 PDMS, Trajectory AE 64.9, and raw trajectories 55.0. The paper reports a 5.1-point gain over AE using unrounded scores.
Panels (a–b) evaluate the final policy separately: one evaluation takes 149 ms versus 490 ms for ten, a 3.29× speedup. Timing includes image encoding and residual correction at batch size one on an RTX 4090.
VITAL rollouts
One-step predictions, scene by scene.
These six browser-playable clips show dynamic RGB, trajectory overlays, fixed-scale BEV, and per-frame diagnostics. Each contains 24 independent NFE = 1 predictions.
Scene 01
One-step rollout
Mean ADE: 0.583 m.
Scene 02
One-step rollout
Mean ADE: 0.500 m.
Scene 03
One-step rollout
Mean ADE: 0.462 m.
Scene 04
One-step rollout
Mean ADE: 0.316 m.
Scene 05
One-step rollout
Mean ADE: 0.452 m.
Scene 06
One-step rollout
Mean ADE: 0.396 m.
Residual refinement
Small corrections, visible effects.
The paper’s one-step before/after examples show improved progress (a), restored drivable-area compliance (b), and improved time-to-collision (c). Panel (b) uses a maximum displacement of 7.9 cm and raises that scene’s PDMS from 0.0 to 100.0.
The action generator and codec stay frozen. A deterministic, bounded correction updates only residual coordinates. These examples illustrate local improvements; the method does not guarantee better reward or safety for every correction.
Current paper · One-step trajectories before/after refinement
Reference
Citation
A project citation is provided below. The arXiv identifier will be added when available.
@misc{liang2026vital,
title = {VITAL: From Visual Motion to Precise Action},
author = {Alan Liang and Youquan Liu and Mingxiao Li and Wei Chow and Lingdong Kong and Yue Liao and Dongyue Lu and Yu Yang and Wei Yin and Runnan Chen},
year = {2026},
url = {https://worldbench.github.io/vital.html}
}