NeurIPS 2026
Spatial Harness
Grounding Spatial Reasoning in Executable World States with Physics Engines
Live engine · a MindCube question from the paper
Execute the turn
Replayed from the paper's run on this question. The Perceiver turns three photos and the question into a world state; the Reasoner then answers by writing Python against the engine, which computes what the camera sees. The replay loops; click any call to take over the camera.
QuestionStarting from View 1, turn 180°. What is behind you after the turn?
- 1Perceiver · builds the world state from the photos and the question
- 2Reasoner · answers by executing calls in MuJoCo
camera [4, 4] · yaw 90° (up)
Replayed from the paper's run on MindCube item rotation_group020_q1_7 with Qwen3.5-27B. Perceiver lines are excerpts of its output; the engine responses and the three MuJoCo images are the originals. Your own calls use a simplified renderer. The grid is a MindCube cognitive map: 10 × 10 cells of 0.919 m, x to the right, y downward.
Abstract
From imagined turns to executed ones
Spatial reasoning requires understanding how objects and observers are arranged in a scene and how their relations change with viewpoint or movement. Vision-language models (VLMs) typically perform both steps through model-internal reasoning, forming an implicit scene interpretation from partial views and imagining the consequences of spatial transformations. A misread scene can lead to a wrong answer, while even a correct interpretation does not ensure that imagined transformations satisfy spatial constraints.
We introduce Spatial Harness, which represents scenes as explicit and executable world states to ground spatial reasoning in physics-engine feedback. A Perceiver integrates scene constraints extracted from the question with specialized visual estimates to place objects and cameras in a shared coordinate frame. The resulting state is instantiated in a physics engine, where a Reasoner executes viewpoint changes and uses computed measurements and rendered views to guide reasoning.
We evaluate on MindCube, SAT, MMSI-Bench, and SpatialRGPT-Bench with open- and closed-weight VLMs. On MindCube, Spatial Harness improves Qwen3.5-27B from 52.70% to 71.14%, surpassing pySpatial by 7.52 percentage points. Given identical annotated object and camera poses, it reaches 98.25% versus 71.81% for model-internal reasoning.
Perceive once, then compute
Spatial Harness separates perception from reasoning. Both run on the same VLM: the Perceiver equips it with perception models, the Reasoner with a physics engine. Every spatial judgment, such as what a turn brings into view, is computed by the engine from the state.
-
in images x, question q → out ŝ
Perceiver
Parses the question into entities, relations, viewpoints and query needs. Runs detection (LocateAnything-3B), depth (MoGe-2), orientation (Orient Anything V2) and relative camera pose (Reloc3r) on every view, then aligns the camera frames, fuses the observations and enforces the stated constraints.
-
ŝ = (ℒ, Ω, 𝒞)
World state
A shared layout ℒ with the position and heading of every entity and camera, the per-view observations Ω, and the constraints 𝒞 stated in the question. The engine is used only when the state holds every entity and camera pose the answer needs; otherwise the VLM answers from the images.
-
in x, q, ŝ, ℰ[ŝ] → out ŷ
Reasoner
Instantiates ℒ in MuJoCo as labeled proxies and writes Python calls that move the camera, measure distances and render views. Each result enters its context and decides the next call, for up to five steps; an answer counts only after at least one engine call.
mujoco_sim calls control the camera pose, return geometric measurements and render views of the same scene. The pose persists across calls.
Four benchmarks · seven VLMs
Results
MindCube has 1,050 multiple-choice questions over a few real views, grouped by camera motion into Rotation, Among and Around. We also evaluate on SAT, MMSI-Bench and SpatialRGPT-Bench. All methods in a comparison use the same VLM.
71.14%
MindCube, from images and question
Base 52.70 · pySpatial 63.62
98.25%
MindCube, same GT map for both
Base with the map 71.81
83.31%
SpatialRGPT-Bench
Base 48.20 · pySpatial 39.79
Same layout, different reasoning
MindCube accuracy (%). Both methods receive the identical annotated map of object and camera poses.
Show the numbers
Against tool-using baselines on MindCube
End-to-end accuracy (%). MindJourney answers from views generated by a video world model; pySpatial writes Python programs over a 3D reconstruction. Bold: best per model and column.
Three more benchmarks
End-to-end results (%). SAT (500 synthetic questions) and MMSI-Bench report accuracy; SpatialRGPT-Bench reports the share of numerical estimates within its error tolerance. Avg is the mean of the three. Bold: best per model and column.
Where the gains come from
Ablations on MindCube with Qwen3.5-27B, overall accuracy (%).
Engine feedback
Same GT map in every condition; only the Reasoner's feedback changes. Extra turns alone add 6.00 points, engine feedback adds 26.44.
Building the world state
The Reasoner is fixed; only the source of the map changes. Both perception channels matter, the visual channel most.
Citation
@inproceedings{cao2026spatialharness,
title={Spatial Harness: Grounding Spatial Reasoning in Executable World States with Physics Engines},
author={Chentao Cao and Zhanke Zhou and Sangni Duan and Bo Han and Hang Li},
booktitle={NeurIPS},
year={2026},
}