GrndCtrl: Grounding World Models via Self-Supervised Reward Alignment

Haoyang He^1,2,Jay Patrikar²,Dong-Ki Kim²,Max Smith²,Daniel McGann²,
Ali-akbar Agha-mohammadi²,Shayegan Omidshafiei²,Sebastian Scherer^1,2

arXiv Page

GitHub Repo

Abstract

Recent advances in video world modeling have enabled large-scale generative models to simulate embodied environments with high visual fidelity, providing strong priors for prediction, planning, and control. Yet, despite their realism, these models often lack geometric grounding, limiting their use in navigation tasks that require spatial coherence and long-horizon stability. We introduce Reinforcement Learning with World Grounding (RLWG), a self-supervised post-training framework that aligns pretrained world models with a physically verifiable structure through geometric and perceptual rewards. Analogous to reinforcement learning from verifiable feedback (RLVR) in language models, RLWG can use multiple rewards that measure pose cycle-consistency, depth reprojection, and temporal coherence. We instantiate this framework with GrndCtrl, a reward-aligned adaptation method based on Group Relative Policy Optimization (GRPO), yielding world models that maintain stable trajectories, consistent geometry, and reliable rollouts for embodied navigation. Like post-training alignment in large language models, GrndCtrl leverages verifiable rewards to bridge generative pretraining and grounded behavior, achieving superior spatial coherence and navigation stability over supervised fine-tuning in outdoor environments.

Overview

Reinforcement Learning with World Grounding (RLWG) addresses geometric inconsistencies in pretrained video world models through self-supervised post-training with verifiable rewards. Instead of reconstruction losses, RLWG grounds models using geometric and perceptual rewards from frozen evaluators.

GrndCtrl instantiates RLWG using Group Relative Policy Optimization (GRPO), enabling physically consistent rollouts essential for reliable world generation.

Problem

Despite impressive generative fidelity, current video world models often capture the appearance of motion more than its structure. Their rollouts remain visually plausible but geometrically and temporally inconsistent: poses drift, depths wobble, and trajectories lose alignment over time.

These instabilities limit the use of current models for closed-loop tasks such as localization, mapping, and planning, where physically consistent representation is essential.

Solution

RLWG refines pretrained world models using verifiable geometric and perceptual rewards derived from model rollouts. Each rollout is automatically scored using rewards that quantify spatial and temporal coherence, such as pose cycle-consistency, depth reprojection agreement, and action adherence.

GrndCtrl uses GRPO to optimize these verifiable rewards efficiently, preserving visual quality while progressively aligning the model's dynamics with measurable structure in the real world.

Qualitative Comparison

Click to see comparisons of GrndCtrl with ground truth and baseline methods across different scenarios.

Method

Figure 1. Overview of GrndCtrl. RLWG refines pretrained world models using verifiable geometric and perceptual rewards. GrndCtrl instantiates RLWG using Group Relative Policy Optimization (GRPO) to optimize these rewards, enabling physically consistent rollouts.

Quantitative Results

Method	Seen				Counterfactual				Unseen
	T↓	R↓	V↑	DTRI↑	T↓	R↓	V↑	DTRI↑	T↓	R↓	V↑	DTRI↑
CODa
Baseline	57.8	1.77	7.40	38.9	71.5	1.55	7.41	39.1	56.9	1.71	7.40	38.3
+T+R	46.4	1.44	7.32	38.4	50.5	1.53	7.34	38.7	54.3	1.75	7.36	39.3
+T+R+DTRI	65.7	1.74	7.43	37.0	57.7	1.86	7.42	36.8	42.6	1.74	7.40	37.1
+T+R+DTRI+V	39.9	1.27	7.35	37.5	40.7	1.42	7.34	37.4	31.0	1.53	7.37	38.0
SCAND
Baseline	186.3	3.76	7.16	23.6	315.9	4.24	7.13	21.4	117.0	4.02	6.99	18.4
+T+R	158.2	3.61	7.19	23.7	251.2	4.34	7.18	21.7	131.1	3.95	7.04	19.1
+T+R+DTRI	157.9	3.65	7.10	22.1	288.6	4.45	7.17	20.1	118.6	4.07	7.03	17.9
+T+R+DTRI+V	133.4	3.30	7.11	24.5	220.1	4.23	7.08	22.8	123.4	3.62	6.98	19.4
CityWalk
Baseline	11.7	3.13	7.96	46.9	13.1	3.27	7.94	47.4	20.8	4.47	7.90	44.5
+T+R	8.9	3.31	7.90	44.9	4.8	4.42	7.91	45.6	10.2	3.47	7.87	42.8
+T+R+DTRI	8.4	3.36	7.84	43.5	4.7	4.40	7.83	44.1	10.9	3.68	7.79	41.4
+T+R+DTRI+V	8.8	3.37	7.84	42.6	4.7	4.37	7.85	43.3	9.9	3.74	7.80	40.8

Table 1: Quantitative evaluation across three datasets (CODa, SCAND, CityWalk) and three regimes: Seen, Counterfactual, and Unseen. We compare baseline against progressive reward combinations (T+R, T+R+DTRI, T+R+DTRI+V). GrndCtrl achieves substantial improvements. Metrics: T (Translation Error, m), R (Rotation Error, rad), V (Video Quality), DTRI (Depth Temporal Reprojection Inliers).

Method	Seen		Counterfactual		Unseen
	T↓	R↓	T↓	R↓	T↓	R↓
Baseline	73.2 ± 243.7	2.38 ± 3.88	75.8 ± 253.9	2.38 ± 3.90	71.2 ± 251.2	2.88 ± 4.28
GrndCtrl T+R 100	72.0 ± 283.6	1.95 ± 3.38	75.9 ± 311.7	1.85 ± 3.22	58.4 ± 231.1	2.57 ± 4.08
GrndCtrl T+R 150	26.7 ± 101.5	1.54 ± 2.84	24.8 ± 99.1	1.53 ± 2.80	26.5 ± 104.3	2.08 ± 3.33
GrndCtrl T+R 200	18.4 ± 68.1	1.40 ± 2.49	16.8 ± 63.0	1.36 ± 2.57	16.3 ± 56.5	1.97 ± 3.11

Table 2: Reliability analysis showing error statistics (mean ± standard deviation) across multiple stochastic rollouts for different GRPO iterations. Baseline exhibits high variance, while GRPO training progressively reduces both mean errors and variance, achieving consistent rollouts. Metrics: T (Translation Error, m), R (Rotation Error, rad).

Citation

@misc{he2025grndctrlgroundingworldmodels,
      title={GrndCtrl: Grounding World Models via Self-Supervised Reward Alignment}, 
      author={Haoyang He and Jay Patrikar and Dong-Ki Kim and Max Smith and Daniel McGann 
              and Ali-akbar Agha-mohammadi and Shayegan Omidshafiei and Sebastian Scherer},
      year={2025},
      eprint={2512.01952},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.01952}, 
}