# Praxis-1: a robot policy that acts on a video it never finishes imagining > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.satyajit.si/articles/praxis-1 > date: 2026-11-12 > tags: robotics, world-models, flow-matching, diffusion, mixture-of-experts On 2026-11-11 Runway released Praxis-1, which it calls a world action model, with open weights under Apache 2.0. There are three policies on Hugging Face, a [base](https://huggingface.co/runwayml/praxis-1-base), a single-arm [DROID policy](https://huggingface.co/runwayml/praxis-1-droid) and a bimanual [YAM policy](https://huggingface.co/runwayml/praxis-1-yam), plus NVFP4 and LeRobot variants. Inference and fine-tuning code is at [`runwayml/praxis-1-oss`](https://github.com/runwayml/praxis-1-oss), and a long [technical report](https://runway.com/research/praxis-1-video-pretraining-to-robot-control) stands in for a paper. The launch thread on X made four claims: first place on NVIDIA's RoboLab-120 leaderboard, ahead of every open VLA including π0.5 and GR00T N1.6; 15 of 15 simulated trials on a Noble Machines robot after under 30 minutes of teleoperated demonstrations; and a run 15.6x faster than GPT-6 Astra driving the same arm, at 51x lower cost per completed task. For this piece I read every safetensors header of the DROID checkpoint by HTTP range request and summed the shapes. I read the sampler, the model and the server code at commit `68f4d7a` (2026-11-11), the configs, and NVIDIA's NeMo fine-tuning tutorial. I also read the live RoboLab leaderboard, and I went back to the launch thread's videos. I ran no model. Nothing here was served on a GPU. The finding that changed how I read the whole release is in the sampler, so the mechanism comes first. ## Three ways to turn pixels into motor commands A robot **policy** maps what the robot sees and feels to what it does next. Modern ones emit an **action chunk**, a short trajectory of future commands executed before the policy looks again. One expensive network call is then spread over many control ticks. Three families compete for that job. A **VLA** (vision-language-action model) starts from a vision-language model and adds a head that outputs actions. π0.5 and GR00T N1.6 are the open reference points. The language model brings semantics: it knows what "the purple comb" is. It brings nothing about how a comb moves when pushed, because no text corpus teaches contact dynamics. A **world model** predicts the future of a scene given an action. It is a simulator you can roll policies through. Runway uses one that way for policy evaluation, and its [September post](https://runway.com/research/introducing-praxis-1) claims a 0.95 correlation between success inside its world model and success on hardware. A world model does not pick actions. A **world action model** (WAM) predicts the future frames and the actions together. The bet is that the same weights that must imagine what the scene looks like after the gripper closes will learn which motion gets it there. Video is cheap and abundant; teleoperation is neither. Runway's September post puts a number on the bet: fine-tuned placement error of 16.1 cm after pretraining on web video against 16.0 cm after pretraining on the same hours of teleoperated robot video, inside one standard error over 93 evaluation pairs. If that holds, the bottleneck on a policy becomes how much ordinary video a model can digest. This site has covered three other WAMs: [FLUX 3 Action](/articles/flux-3-action), NVIDIA's [Cosmos 3](/articles/cosmos-world-model) policy variants and [Flex-π](/articles/flex-pi). The cost of a WAM is tokens. Video is thousands of them per second, actions a few dozen. Every design in this family is an answer to the question of how much imagined video you pay for per action, and when. ## What is in the 54.4 GB Praxis-1 has three parts. A video backbone predicts the next stretch of video. A small **action expert** reads the backbone's hidden states and predicts the motion. Thin per-robot projections translate each robot's poses and actions into the shared format. Runway's architecture figure:
The weights tell the same story in more detail. Summing all 567 tensor shapes in the 17 shards of `praxis-1-droid` gives 27,202,371,308 parameters. Hugging Face's API says 27,202,371,288. The gap of 20 is the twenty scalar tensors of shape `[]`, one per backbone block (I counted them), which the API skips. The file total is 54,404,746,968 bytes, all BF16 except 2,176 FP32 values. | Component (DROID checkpoint) | Parameters, summed from headers | |---|---:| | Backbone encoder, 14 blocks | 18.905B | | Backbone decoder, 6 blocks | 7.983B | | Action expert, 16 layers plus heads | 314.4M | | **Total** | **27.202B** | | of which routed MoE experts | 25.669B (94.4%) | The backbone is a diffusion transformer split into a condition encoder and a velocity decoder, the DDT layout of [Wang et al.](https://arxiv.org/abs/2504.05741). It works on latents of a frozen Wan2.2 VAE and reads instructions through a frozen Qwen3.5-2B. Seventeen of its 20 feed-forward layers are mixture-of-experts. Each MoE layer has one shared expert and 128 routed experts, and every token takes the shared one plus eight routed ones, balanced the [DeepSeek-V3](https://arxiv.org/abs/2412.19437) way. The config shows why the report can say per-token compute "stays about the same as a dense block". Every expert has a hidden width of 1,280, and nine of them make 11,520, which is exactly `dense_ffn_hidden`, the width of the three dense layers. A token pays for one dense FFN's worth of matrix multiplies while the layer stores 129 experts. Applying 8 of 128 to my routed total gives 2.823B active parameters in the backbone, which matches the report's 2.8B. ### Tokens per chunk Time is cut into **chunks** of 1.6 s: six latent frames, 24 frames at 15 fps, 24 action rows at 15 Hz. The DROID config fixes the shapes, and multiplying them out shows where the compute goes. Two cameras, exterior and wrist, are resized to 224×416 tiles and placed side by side on a 224×832 canvas. The VAE compresses 16× in each spatial direction, giving a 14×52 latent with 48 channels, and the backbone patchifies 2×2. That is 7×26 = 182 tokens per latent frame and **1,092 video tokens per chunk**. On the action side, the expert holds one pose token, eight learned "future" tokens and six action tokens per chunk. Each action token packs four rows of the 10-D action (`action_patch_dim: 40`). Fifteen expert tokens against 1,092 video tokens: the arm's motion is a rounding error in the sequence. An action row for the Franka is 10 numbers: xyz, a 6-D rotation and the gripper. Position and rotation are relative to the pose at the start of the chunk. Position and gripper are rescaled to [-1, 1] by the 1st and 99th percentiles of DROID, and the exact bounds are written into `config.json`, so no dataset statistics are needed at inference. ### Chunk-causal attention The base video model generated whole clips with every frame seeing every other, including later ones. A robot cannot see its own future, so mid-training switches the attention to **chunk-causal**: chunk $k$ attends to itself and to clean copies of chunks before it. In training, those earlier chunks are real video (teacher forcing), which lets Runway train four chunks in one parallel pass over a 93-frame window. The action expert gets a mask of its own:
That last column is the whole idea. The actions for the next 1.6 s are computed by reading eight layers of the backbone while it denoises its picture of the same 1.6 s. ## Joint denoising, and what "jointly" means here Both streams are trained with rectified flow. Take a clean sample $x_0$ (a chunk of video latents, or a chunk of action rows), Gaussian noise $\varepsilon$ and a time $t \in [0, 1]$, and mix them along a straight line: $$ x_t = (1-t)\,x_0 + t\,\varepsilon, \qquad v = \varepsilon - x_0 . $$ The network $v_\theta(x_t, t, c)$ learns the velocity $v$ given conditioning $c$: the instruction, the clean context frames, the pose. Runway's loss is an equal-weight sum of the video and action velocity losses. Each chunk draws its own video noise level, weighted toward high noise, and its own action noise level, uniform and independent. The independence matters: the expert learns to read the backbone when the imagined video is nearly clean, when it is nearly pure noise, and everywhere between, regardless of how noisy its own actions are. At inference the sampler walks $t$ from 1 to 0 with Euler steps, $$ x_{t'} = x_t + (t' - t)\,v_\theta(x_t, t, c), $$ and the clean-sample prediction the network implies at any point is $\hat{x}_0 = x_t - t\,v_\theta$. In the released code, "jointly" runs in one direction only. The docstring of `plan_window_joint` says it outright: "No trunk pass reads the actions or the proprio: the video Euler state moves with the video velocity only." The video stream is denoised as if no actions existed. The action stream is denoised by the expert, which reads the backbone's taps at each step. Video shapes actions, and never the reverse. During training, the action loss does push gradients into the backbone through those cross-attention reads, so the backbone learns features the expert finds useful. At inference the backbone never sees the expert's output. ## Two steps, and a video that never finishes The shipped settings are in `configs/serve_droid.yaml` and `configs/sim_droid.yaml`: `num_steps: 2`, `t_shift: 12.5`, `cfg_video_scale: 4.0`, `cfg_scale: 1.0`. The time grid comes from `window_time_grid` in `praxis/sampler.py`: $$ u_j = 1 - \frac{j}{N}, \qquad t_j = \frac{s\,u_j}{1 + (s-1)\,u_j}. $$ With $N = 2$ and $s = 12.5$, $u = (1, 0.5, 0)$ becomes $t = (1, 0.926, 0)$. So the model is evaluated twice: once on pure noise, once at $t = 0.926$, where the latent is 92.6% noise by mixing weight. The first Euler step moves the state 7.4% of the way. The second covers the remaining 92.6% in one jump. That jump has a useful property. The last Euler step goes from $t$ to 0, so $$ x_0 = x_t - t\,v_\theta(x_t, t, c) = \hat{x}_0(x_t, t, c). $$ The output of the sampler is exactly the network's clean prediction made at its last evaluation. For Praxis-1 that means the 24 action rows the robot executes are the action expert's one-shot guess, made while it reads a backbone whose video latent is still 92.6% noise. Runway's report describes the knob in softer words: "A larger shift leaves more of the denoising to the second step." Then there is `skip_video_decoder: on`, set in both serve configs. The server code explains it: the last step "moves only the actions, which read only the encoder taps", so the video decoder is skipped and "the plan has no video (None)". Counting the passes in `plan_window_joint` for one served DROID call: 1. Step 1 at $t = 1$: encoder and decoder with the instruction, encoder and decoder with a blank caption (video guidance 4 needs both), one Euler step on the video latent. 2. Step 2 at $t = 0.926$: the encoder only, with the instruction. No decoder, no unguided pass. 3. Two passes of the action expert, one on each step's taps. Three encoder passes, two decoder passes, two expert passes. The robot moves on actions read from a future that the model starts imagining and abandons 7.4% of the way in. The "predicted chunk of future video" in the report's prose exists on the server only when you turn the flag off; it is not in the default response. The README is plain about this, so it is not a hidden trick. It does mean the "world" half of this world action model is, at serving time, a set of hidden states. The video guidance still matters, through a narrow channel. At step 1 the backbone sees only noise plus context, so its taps depend on the context and the instruction. Guidance acts on the video velocity of step 1, and that guided velocity moves the latent the 7.4% that the step-2 encoder pass then reads. Runway's guidance search (section 9 of the report) found that adding video guidance and raising the shift together ended 4.0 points above its start of no guidance at shift 8. Pushing either further scored lower. The report says the search ran on "a fixed set of simulated environments" and does not say which; I could not tell from the repository whether those were RoboLab tasks. Here is the sampler on a toy problem. The time grid is Praxis-1's. The action distribution is mine: a 24-row reach that should go to one of two mugs, with an exact denoiser for that two-way choice and a slider for how strongly the context favours the left one. With 8 steps and no shift, the sampler resolves the choice by itself: by $t \approx 0.25$ the noise is small enough that the partial chunk already looks like one reach or the other, and the posterior collapses. With Praxis-1's 2 steps at shift 12.5, the last read is at 0.926, the partial chunk is almost all noise, the posterior still equals the context prior, and the output is the prior-weighted average of the two reaches. At 50/50 that is a reach for the empty space between the mugs. The toy can't speak for the learned model, whose conditioning makes most scenes unambiguous. What it shows is where the choice has to come from at these settings: the context frames, the instruction and the barely started imagination, not the iteration. Runway's own error analysis points the same way. On 32 held-out DROID windows, the mean end-effector error is 5.71 cm when the model samples its own video and actions, and 1.14 cm when it is handed the true future video, noised to the sampler's level, and samples only the actions.
And the limitations section names the most common real-robot failure as "reaching for an object that resembles the target, and the wrong choice is usually already visible in the imagined video". A look-alike mug is exactly the case where the context doesn't settle the choice. ## The control loop, 1.6 seconds at a time Each call returns 24 rows (21 on the first call of a context window, whose first latent frame covers one frame instead of four). In the default **chunked** schedule the robot executes all of them, sends back the frames it saw over one WebSocket, and calls again. The report gives p50 server times on one H100 of 522 ms for the DROID policy and 545 ms for the bimanual one, and about 0.55 to 0.6 s round trip. Waiting is the price. If a call takes $L$ seconds and a chunk lasts $H = 1.6$ s, a synchronous loop holds still for $L / (L + H)$ of the time. The report measured that on its bimanual station: 32% holding still when calls took 0.8 to 0.9 s, and 16% at about 0.3 s. The formula gives 33.3 to 36.0% and 15.8%. Close enough that the loop is doing what it says. **Real-time chunking** (RTC), opt-in in the repo, asks for the next chunk before the current one ends. The hard part is the junction. Runway's blended plans made the arm oscillate, so the new plan starts from where the arm will be. On the bimanual station RTC cut holding still to 5% and 3%. On a five-object pick-and-place into a bag it finished faster (median 9.1 s against 15.2 s) at 24 of 30 successes against 28 of 30 chunked. With streaming optimizations the p50 call drops to 124 ms (DROID) and 128 ms (YAM). The slider exposes the other cost of chunking: reaction time. A cube that slips just after a call took its frames is first seen by the next call, a full chunk plus one call later, and acted on one call after that. At 545 ms that is 2.69 s in the worst case. RTC shortens it by one call, not by a chunk; the chunk length itself is what bounds how fast this policy can notice anything. ## Four stages of training Runway trains the world before the actions. Each stage starts from the one before, and the video loss stays on throughout. Stage one is Runway's video foundation model, trained on twelve generation tasks with full-sequence attention. Stage two switches to chunk-causal attention on embodied video at 15 fps: 58% egocentric human video, 20% general video, 18% real-robot video and under 1% gameplay, by tokens. Stage three attaches the expert with fresh weights and adds action targets for single arms, arm pairs, handheld grippers and human hands, ramping the action gradient into the backbone from zero over 1,500 updates and keeping about 30% of tasks video-only. Stage four fine-tunes for one robot; for DROID that is about 277 hours of successful episodes, 128 H100s and 50k steps. The fine-tune is a full fine-tune of 27.2B parameters. The per-robot projections are the only new weights, but every backbone matrix moves. NVIDIA's NeMo tutorial does the same on one 8-GPU node, with the 128 routed experts split 16 per GPU so the FP32 optimizer state fits. Runway's compute-scaling test is the strongest evidence that mid-training matters. Five DROID fine-tunes from successive mid-training checkpoints scored 25.5%, 35.2%, 39.2%, 41.2% and 41.2% on RoboLab-120. On RT-1, a robot absent from all mid-training, five fine-tunes rose from 56.3% to 80.0% on SimplerEnv, though not monotonically (65.6%, 66.9%, 64.1% between). Runway calls its fitted line "a guide rather than a law", which is the right amount of confidence for five points. The released DROID policy scores 46.2%, not 41.2%; the guidance and shift search accounts for some of the gap, and the report does not reconcile the rest. ## The launch claims, one by one ### "#1 on NVIDIA's RoboLab-120 leaderboard" [RoboLab-120](https://research.nvidia.com/labs/srl/projects/robolab/leaderboard.html) is NVIDIA's Isaac Sim benchmark of 120 tabletop tasks for a DROID-style Franka, 10 trials per task. Here are selected rows of the unfiltered board as I read it on 2026-11-12 (Cosmic, tied sixth at 42.3%, and HiDream-O1-Embodied at 39.9% are among those left out), with exact 95% Clopper–Pearson intervals I computed from the success counts: | # | Policy | Type | Successes | Rate | 95% interval | |---:|---|---|---:|---:|---| | 1 | AMP1 | agentic LM | 2,579 / 3,000 | 86.0% | | | 2 | SenseNova-RoboRSI | agentic LM | 685 / 1,200 | 57.1% | | | 3 | Lakeside | LWM | 1,531 / 3,000 | 51.0% | | | 4 | BiMind v0.5 | VLA | 590 / 1,200 | 49.2% | | | 5 | **Praxis-1** | WAM | **555 / 1,200** | **46.2%** | 43.4–49.1% | | 6 | FLUX 3 Action | WAM | 515 / 1,200 | 42.9% | 40.1–45.8% | | 7 | Atomic-WAM | VLM+WAM | 475 / 1,200 | 39.6% | | | 8 | Cosmos3-Nano-Policy | WAM | 441 / 1,200 | 36.8% | | | 11 | π0.5 | VLA | 336 / 1,200 | 28.0% | | | 15 | GR00T N1.6 | VLA | 87 / 1,200 | 7.2% | | Praxis-1 is fifth. The report says so itself, in a paragraph the tweet left out: the top two are frontier models driven as agents, and Lakeside and BiMind v0.5 are closed models trained partly on simulation data, BiMind v0.5 fine-tuned on RoboLab data. The link in Runway's post opens the board pre-filtered to robot foundation models with open weights. Under that filter, or among WAMs, Praxis-1 is first. The precise claim is the report's: "#1 world action model" and "#1 open source robot foundation model". "#1 on the leaderboard" is the tweet's compression. Within the claim that does hold: - **It is verified.** The board marks Praxis-1 as checked by the organizers, submitted 2026-11-05. Its detail panel declares no simulation data, where BiMind v0.5's declares RoboLab and Isaac data in fine-tuning. - **It is one run.** There is no seed table. The FLUX 3 Action entry it displaces was the best of six seeds whose mean was 41.60%, as this site [counted in September](/articles/flux-3-action). Against that mean the lead is 4.6 points. Against the single best seed it is 3.3 points, and a two-proportion test on 555 and 515 of 1,200 gives z = 1.64, p = 0.10. Episodes cluster by task, so the true uncertainty is wider still. - **It wins on harder tasks, not easy ones.** On the board's difficulty split, FLUX 3 Action does better on simple tasks (314 of 640 against Praxis-1's 300). Praxis-1 leads on moderate tasks (187 of 390 against 153) and complex ones (68 of 170 against 48). That last gap is the only one of the three a simple two-proportion test calls significant (z = 2.29). Longer tasks are where a chunk-causal video prior should help, so this is the more interesting result, and the launch did not mention it. - **"Ahead of every open-weight VLA" holds** by a wide margin: π0.5 at 28.0%, GR00T N1.6 at 7.2%. Runway's chart prints GR00T as 7.3%; 87 of 1,200 is 7.25%, so that is a rounding direction, not an error.
On real hardware, Positronic ran Praxis-1 on FLUX 3 Action's own ten published Franka tasks, 10 episodes each, in two sessions of five on one rig on November 2. Praxis-1 succeeded 88 of 100 times, FLUX 3 Action 80. The two tie on five of ten tasks; the largest gap is "cup in bowl", 100% against 70%. A two-proportion test on 88 against 80 gives p = 0.12. The report does not say whether the operator was blind to which model ran, which BFL's own Positronic comparison in September did state. ### 80.0% on RT-1, a robot it never saw Fine-tuned for 30k steps (about 28 hours on 32 H100s) on the full RT-1 training split, 78k episodes from the head camera, Praxis-1 scores 80.0% on SimplerEnv's visual-matching grid of 864 episodes over four tasks. The original RT-1 policy scores 52.4%.
Two cautions. RT-1 is a 2022 policy, and much of the 27.6-point gap is one task: "put apple in drawer", 83.3% against 6.5%. On "open / close drawer" Praxis-1 is worse than RT-1, 56.5% against 73.1%. And the 39.7% for FLUX 3 Action is a competitor fine-tuned by Runway with a recipe written for a different robot. That is a reasonable effort, not a tuned baseline. ### Fifteen of fifteen on a Noble Machines robot I could not find this anywhere but the tweet. The technical report does not mention it. The launch post lists Noble Machines only as a launch partner, in a sentence about "a legged humanoid, a six-axis industrial arm and a dual-arm warehouse robot". The September post shows a Noble Machines clip of bimanual manipulation at 2x speed. The tweet's own 21-second video is a grid of simulated runs, each tile badged "Simulation", of a humanoid moving wooden blocks into a bin.