GEM-4D: a video world model taught geometry it never has to output

A video model can show a robot arm picking up a cup and still be no use for picking up the cup. The frames look right one at a time. Between them, the gripper's fingers bend a little, the cup's rim slides a few pixels against the table, and the depth an estimator reads off frame 12 does not agree with frame 13. A person watching does not notice. A program that turns the video into joint angles does, because it has to commit to one 3D position per frame.

GEM-4D, from Kaichen Zhou, Yuzhen Chen and colleagues at Harvard, MIT and Princeton (ECCV 2026), attacks that gap during training only. A second, small transformer is bolted onto the video model's middle layer and made to predict what a frozen geometry model sees in the same clip. Its loss can only fall if that middle layer encodes depth, camera motion and object motion. After training, the second transformer is thrown away. The launch post by @alex_kai2020 puts it as "Train with 3D geometry. No extra video-generation overhead", and quotes seven simulated tasks going from 22.7% to 75.1% success with the same action extractor.

I read the paper (v4, 3 August 2026), the project page, the released code at kaichen-z/GEM-4d and the index of the released checkpoint. I did not train or run anything. The training trick is clean and the code matches the paper's description of it. The headline numbers need more care than the post gives them, and the part that turns video into robot actions is not in the release.

A video model as a robot policy

The recipe GEM-4D builds on goes back to UniPi: do not learn actions directly. Learn to predict the future as video, then read the actions off that video. The paper writes it as two functions. A world model $M$ takes the first frame $\mathbf{I}_0$ and an instruction $c$ and returns frames $\{\mathbf{I}_t\}_{t=1}^{N}$. A policy extractor $P$ takes those frames and returns actions $\{\mathbf{a}_t\}_{t=0}^{N-1}$.

The attraction is data. Robot demonstrations with actions are scarce and tied to one arm. Video of things being manipulated is plentiful, and a pretrained video generator already knows a lot about how objects move. The action-specific part, $P$, can be a fixed pipeline of off-the-shelf vision models with no training per task.

That differs from the world action models this site has covered recently. Praxis-1 and FLUX 3 Action put an action head inside the video model and denoise actions in the same pass, trained on robot data, and Flex-π denoises 3D pointmaps and actions in one shared latent. Vision-language-action models skip the video and map pixels and text straight to motor commands. GEM-4D keeps the two stages apart, so the video has to be good enough for a separate program to measure. Whether it is depends on a property the training loss of a video model never asks for.

Why pixels cannot pin down geometry

The paper's Section 3.2 starts from where a scene point lands in the next frame. A point $\mathbf{X}_t$ seen at pixel $\mathbf{p}_t$, at depth $\mathbf{D}$, with the camera moving by rotation $\mathbf{R}$ and translation $\mathbf{T}$ and the point itself moving by scene flow $\Delta\mathbf{X}$, projects to

where $\mathbf{K}$ is the camera's intrinsic matrix. Four unknowns, $(\mathbf{D}, \mathbf{R}, \mathbf{T}, \Delta\mathbf{X})$, decide every correspondence between frames. The paper draws two conclusions from it.

The first is that a pixel loss cannot enforce correspondence. Many settings of the four unknowns produce the same pixels: double the depth and double the motion and nothing on screen changes; pan the camera left while the object stays still, and it looks like the object moved right. The paper's sentence is blunt: "A pixel loss can reach zero while the underlying correspondences are entirely wrong." A video diffusion model trained on pixels or VAE latents is graded on what is on screen, so nothing stops it from implying one depth in one frame and another in the next.

The second is the reverse. A representation that does encode the four unknowns has no freedom left, because the equation then fixes where every point goes.

The widget makes the first point concrete in one dimension. The left panel is fixed: five frames in which the gripper moves the same number of pixels each step. The right panel is what those frames mean in 3D under different choices of depth and camera motion, lifted the way an action extractor lifts them, with depth times the ray through the pixel.

Scaling the depth stretches the reach, and shifting motion onto the camera shrinks how far the gripper itself travels; the frames never change. The drift toggle is the failure that matters for a robot. Each frame is lifted with its own depth, as happens when a generator's geometry is not consistent across frames, and a straight reach becomes a zigzag. An extractor that follows it will drive the arm along the zigzag.

Teaching the backbone through a branch it will lose

Earlier fixes for this changed what the video model outputs. TesserAct generates RGB, depth and surface normals together; 3DFlowAction predicts 3D flow. GEM-4D's argument against that route is that it needs geometric labels at scale and constrains the output space of a pretrained backbone.

GEM-4D leaves the output alone and works on the inside. Feed-forward geometry models in the DUSt3R and VGGT line, and Depth Anything 3, are trained to estimate per-frame depth and camera pose across a clip. Those are the dominant factors in the equation above. The paper's claim is that their intermediate features therefore encode the correspondence structure, including object motion, since a point whose depth changes in a way the camera motion does not explain must be moving. Making the video model predict those features should push the same structure into it.

expand

Both branches are flow matching. The video branch is a standard latent video DiT: noise $\mathbf{z}_1$ and a clean latent $\mathbf{z}_0$, a velocity field trained against the analytic target,

The geometry branch runs the same objective on a different latent. A frozen geometry model $G$ turns the clip into $\mathbf{g}_0$, which is noised to $\mathbf{g}_t$, half its tokens are masked, and a Geometry DiT predicts the velocity, conditioned on $\mathbf{m}_t$, the video DiT's features at one middle block:

The design choice that makes this work is in what the Geometry DiT can see. It gets the masked, noised geometry latent and $\mathbf{m}_t$, and nothing else from the scene: no pixels, no depth, no camera. To predict how the geometry latent should be denoised, the information has to come through $\mathbf{m}_t$, and the gradient of the geometry loss flows back into the video backbone through it. The coupling runs one way. The Geometry DiT reads the video features and never writes back, so the video DiT runs alone at inference with the same weights layout and the same cost as before.

This is close in spirit to REPA, which aligns a diffusion model's hidden states with a frozen encoder's features through a projection. The difference is that GEM-4D's target is a whole generative problem rather than a cosine similarity: the branch has to denoise the teacher's features, not just point in the same direction. Geometry Forcing, the closest baseline in the paper, is the REPA-style version with a geometry teacher.

What the code actually trains

The paper does not give $\alpha$, which block $\mathbf{m}_t$ comes from, or which geometry model is the teacher for the main results. It lists PAGE-4D, Depth Anything 3, VGGT and the DUSt3R family as examples. The released code fixes all of it, in training/config/default.yaml and model/gem4d/:

geometry:
  teacher: weights/da3            # DepthAnything-3 (local dir or depth-anything/DA3-LARGE-1.1)
  feat_layer: 12                  # DA3 ViT-L layer used as the geometry representation g0
  depth: 4                        # Geometry DiT blocks
  mask_ratio: 0.5                 # fraction of geometry tokens masked (mask mechanism M)
  alpha: 0.03                     # weight of L_geo
  coupled_noise: true             # geometry and video share one noise sample

A few details in the code are not in the paper and change how I read the method.

• The teacher is DA3-Large, layer 12, through a frozen random projection. GeometryProjector compresses the 1,024-channel features to 256 with a Xavier-initialised 1x1 convolution, box-averages every 4 frames (49 frames become 13, matching the VAE's temporal stride), resizes to the latent grid, and projects to the latent's channel count (16 on Wan-14B). All of it has requires_grad_(False). The geometry latent is a fixed random view of DA3's features, shaped like a video latent.
• The noise is shared. With coupled_noise: true the geometry latent is noised with the same sample as the video latent, so both branches denoise the same random draw at the same timestep.
• The Geometry DiT reads $\mathbf{m}_t$ only through cross-attention. Each of its 4 blocks is self-attention over geometry tokens, cross-attention to the video features, and an MLP, with zero-initialised gates so the branch starts as the identity. On the Wan-14B config hook_layer: 20 of 40 blocks; on Wan-5B it is block 15 of 30. The hook does not detach the features, so the gradient reaches the backbone.
• The weight is small. $\alpha$ is 0.03, ramped in linearly over the first 100 steps.

What "no extra overhead" covers

At inference, the claim holds exactly. inference/infer.py loads only the video DiT. The released checkpoint's index lists 1,303 tensors and a total of 32,789,757,568 bytes; in bf16 that is 16.39B parameters, and none of the tensor names contain geo. The deployed model is a plain Wan2.1 image-to-video DiT.

Training is another matter. Counting the Geometry DiT from heads/geo_dit.py with the Wan-14B width of 5,120 and 40 heads, I get 1.73B parameters: four blocks at about 419M each, plus the timestep MLP and the in and out projections. Added to the 16.39B backbone that is 18.1B, which matches the README's "18.1 B trainable parameters" for full fine-tuning with the branch. So training carries about 10.6% more trainable weights, plus a DA3-Large forward pass on every 49-frame clip. The README asks for eight GPUs with at least 141 GB each for the 14B run, or optimizer state offloaded to CPU on eight 80 GB cards. On Wan-5B the branch is 0.62B by the same count. That is the right trade for a model you train once and sample many times, and it is still a cost.

From frames to a gripper pose

The second half of the paper is the extractor, the Adaptive Inverse Dynamic System. Its job is to turn a generated clip into a 6-DoF end-effector trajectory and then joint commands. It uses no learned parameters of its own and no per-task training; it chains existing models.

expand

1. 3D grounding. The geometry model supplies depth and intrinsics for the generated frames. Qwen3.5-VL and SAM 2 segment the target object and the end effector from the instruction. FoundationPose aligns a CAD model of the gripper to its point cloud to get the starting pose.
2. Tracking with two triggers. CoTracker3 follows keypoints sampled on the gripper mask. The system watches the fraction of anchors still tracked, $s_t$, and its frame-to-frame change $\Delta s_t$. A slow decay below a threshold means drift: re-sample keypoints from the last good mask. A sudden drop means a generation artifact: ask the vision-language model to find the gripper again.
3. Pose fallback. FoundationPose gives a pose and a confidence per frame. Where the confidence is low, or the pose jumps more than a threshold in translation or rotation, the frame is rejected; translation is taken from the centroid of the gripper's depth pixels and rotation is slerped between the nearest good poses.
4. Grasp and inverse kinematics. GraspGen proposes grasps on the object's point cloud. The one closest, in a weighted sum of translation and rotation error, to the trajectory's pose nearest the object is inserted as a waypoint, the path is smoothed, and inverse kinematics produces the actions.

Read as engineering, this is a robust tracker written to survive bad video. The two-trigger rule and the depth-centroid fallback exist because generated frames have exactly the artifacts the first half of the paper tries to train out. Two things follow. "No extra overhead" is a statement about the generator; the full path from image to motion runs a VLM, SAM 2, CoTracker3, FoundationPose with a CAD model of the gripper, GraspGen and a geometry model on top of 30 sampling steps of a 14B DiT with classifier-free guidance. And the extractor is open-loop: it reads a whole imagined clip and plans from it. The README is explicit that this module is "Not in this repository", along with the VGGT and depth-target ablations.

What the video numbers say

Table 1 scores generated clips on two held-out sets: 400 unseen Droid samples (real) and 780 unseen RLBench samples (simulated), each generated 20 times. Appearance is FVD, SSIM and PSNR. Geometry is AbsRel, $\delta_1$ and $\delta_2$ of depth estimated from the generated frames by Depth Anything 3, a Chamfer distance between point clouds, and $\delta^{vis}_{avg}$, the fraction of visible points tracked within 1 to 16 pixels.

GEM-4D leads every column on both sets. The largest margins are geometric: on RLBench, Chamfer falls from 0.1813 (TesserAct) and 0.1488 (Geometry-Forcing) to 0.0702, and $\delta_1$ on Droid rises from 71.18 (Wan) to 78.19. The paper says the standard deviation across the 20 generations is 1 to 2%, which is smaller than most of these gaps.

expand

Two caveats sit under the geometry columns. On Droid there is no recorded depth, so "ground truth" depth is Depth Anything 3 run on the real frames, and the code's teacher is the same model. A generator trained to encode DA3's features being scored by how DA3 reads its output is not circular in a way that invalidates the result, but it favours this method by construction. RLBench, where depth is the simulator's, does not have that problem, and the gaps there are as large. GEM-4D's RLBench Chamfer is less than half the best baseline's.

The second is the backbone. The paper does not name GEM-4D's own backbone in Table 1; the README says those rows are "GEM-4D on a CogVideoX-5B backbone, not part of this release". So the cleanest comparison in the table is GEM-4D against CogVideoX, same family, fine-tuned on the same training set without the branch, and the gap there (AbsRel 22.33 to 20.13, Chamfer 0.2670 to 0.2001 on Droid) is the effect of the geometry loss plus whatever else differed in the recipe.

The ablation in Table 3 is real-domain only. Predicting depth velocity instead of geometry features (GEM-4D (Dep)) recovers part of the gain: $\delta_1$ 74.60 against 78.19. Swapping in VGGT as the teacher lands below the fine-tuned Wan baseline on SSIM (75.89 against 76.24) and AbsRel (21.73 against 21.39), roughly where plain CogVideoX sits, which the authors attribute to VGGT being trained on static scenes. That result is the most interesting one in the paper to me: the teacher matters, and a weak teacher is not free.

Where 22.7% to 75.1% comes from

The post's headline is not printed in the paper. It is the unweighted mean of seven RLBench columns in Table 2. TesserAct's cells sum to 159, so 159 / 7 = 22.7%; GEM-4D's sum to 526, so 526 / 7 = 75.1%; the difference is 52.4 points. That arithmetic checks out. What the numbers measure needs reading carefully.

The extractor is GEM-4D's own. The paper says it plainly: "As the inverse dynamics module of TesserAct is not open-source, we use our AIDS to process its generated videos." "Same action extractor" is true, and it is the right choice when the alternative is no comparison. The paper does not say how the extractor's thresholds were set, but it was built by the same authors alongside their own model's videos, and I would not assume it is neutral. TesserAct's own pipeline might do better or worse on its videos; nobody has measured it.

One baseline, and it is not the same model minus the branch. CogVideoX would have been the ablation that isolates the geometry loss, and it has no RLBench row: "most videos generated by CogVideoX cannot be reliably processed by our AIDS". That is itself evidence for the paper's thesis, since the extractor fails outright on a plain fine-tune. But it leaves no success number for "same backbone, same data, no geometry branch". The ablation table has no success column either.

The training data is not stated to match. CogVideoX and Wan are described as fine-tuned on the authors' training set (ManiSkill3, RLBench, Bridge, RT-1). TesserAct is described as a CogVideoX model that "further fine-tunes CogVideoX on robotics datasets", with no mention of retraining it on the same set.

No trial counts or spread. Table 2 gives one percentage per task and no number of episodes or variance. Two TesserAct cells, Put Rubbish In Bin at 0 and Reach Target at 2, carry a lot of the mean; leave them out and the gap is 42.4 points over five tasks. That is still large. It is also a sign of how much rides on whether the extractor can lock onto a gripper at all.

The real-world number needs the most correction. The abstract says GEM-4D "improves real-world manipulation success from 61% to 81%". The Metric paragraph of Section 4.2 says the Droid columns come from "a human study to assess the quality of generated videos, focusing on task success, object and robot-arm deformation, and instruction following", averaged over 15 participants. No robot executed those tasks. The three columns average 60.7 and 81.7, so "81" is rounded down. The project page calls it "Real-robot success", which is the wrong label for a rating of videos.

There are real-robot runs: the paper's Figures 4 and 6 and the post's video show a UF arm executing trajectories extracted from generated rollouts. They are qualitative, with no count of attempts.

expand

What was released, and what was not

The repository is more useful than most paper code. It trains GEM-4D on two backbones, Wan2.2-TI2V-5B and Wan2.1-I2V-14B, with RGB-only baseline configs beside them (wan14b_rgb.yaml, wan5b_rgb.yaml), so the missing ablation can at least be run by someone with the GPUs. It ships a 14B checkpoint under Apache-2.0 on Hugging Face, a post-processing tool that estimates depth and scene flow from any generated clip, and CPU smoke tests with tiny random models. The repository itself has no licence file.

It does not reproduce the paper. The checkpoint is a Wan-14B model, while Table 1's rows are a CogVideoX-5B model that is not released. The release's training mix is RLBench, Bridge, RT-1 and Droid at 0.40, 0.25, 0.25 and 0.10; the paper's training list does not include Droid and evaluates on "unseen" Droid samples. And without the action extractor, nothing in the release turns a video into a robot motion, so the RLBench success table cannot be checked from it. Inference alone needs about 48 GB of GPU memory at 240 by 416, 49 frames, per the README.

The idea is worth more than the headline

I like the method. "Make the hidden state predict what a frozen expert sees, through a branch you will delete" is a general move, and here it targets the right failure: video that is plausible frame by frame and incoherent as geometry. It costs nothing at sampling time and about a tenth more parameters at training time, and on RLBench, with simulator depth, the geometry metrics improve by more than the noise. The VGGT ablation adds a useful warning that the choice of teacher can make things worse.

I do not buy "+52.4 pp" as a measure of what geometry supervision does for manipulation. It compares GEM-4D against a different model trained on possibly different data, through an extractor its authors built for their own videos, on seven tasks with no trial counts. The "61% to 81%" real-world claim is a rating of videos by 15 people. The experiment that would settle it is in the repo's configs already: train wan14b_rgb and wan14b_gem4d on the same manifest, run both through one extractor, report episodes. Until that exists, I would cite GEM-4D for Table 1 and the training trick, and treat the success numbers as encouraging.

For a related angle on the same problem, the Cosmos 3 and ABot-World-0 pieces cover world models scored on generation, and SAIL is a policy that searches imagined trajectories in a simulator rather than in video.