METEOR: an agent-built driving network that perceives a lot and plans a little

On 12 November Shinpei Kato, who founded TIER IV and started Autoware, posted a short video and a handful of claims about a model called METEOR. In translation: it is an autonomous-driving model designed and implemented by agentic AI; no human code and no human labels were needed; anyone can build one once the spec is fixed; it has 16 million parameters, so it runs on any SoC; and it is, fairly, an open-source FSD. He posted it as a quote of his own post from the day before about AutoE2E, the Autoware community's end-to-end model, "just under 80 million parameters".

Those are five different kinds of claim. Two are about method (agent-written code, machine labels), one is a number, one is about hardware, and one is a comparison with Tesla's product. I cloned tier4/METEOR at commit cc66a93 and autowarefoundation/auto_e2e at bc7e27c, read both, and read the released checkpoints on Hugging Face without executing them. METEOR is a careful piece of engineering with unusually honest documentation. Most of the post holds up. The number doesn't, and the last sentence is a stretch the repository itself declines to make.

Two ways to get from pixels to a path

A self-driving car has to answer one question many times a second: where should I be over the next few seconds? The classic answer, and the one Autoware is built around, is a modular stack. A localization node matches LiDAR scans against a prebuilt point-cloud map and says where the car is. Perception nodes turn sensor data into a list of objects with boxes, classes and velocities. A prediction node forecasts where each object goes. A planner searches for a trajectory over a lane-level HD map, respecting rules someone wrote down. A controller tracks that trajectory. Each arrow between nodes is a message type that a human designed.

That design is debuggable: when the car brakes for nothing, you open the object list and find the phantom box. Its weakness is the same arrows. An object list drops everything nobody thought to put in it, like a traffic officer's hand signal or a cyclist's lean before a turn, and the planner can only reason about what crosses the interface.

An end-to-end model removes some of those arrows. Cameras go in, a trajectory comes out, and the internal representation is whatever gradient descent finds useful for imitating good drivers. The weakness flips: there is no object list to inspect when it goes wrong, and imitation only teaches the states that human drivers visited, so a model that drifts slightly off a human path is in territory it never saw.

METEOR and AutoE2E sit at different points on that line. The widget lays out what crosses each boundary in the three designs.

METEOR is the more conservative of the two. It is end-to-end in the sense that one network emits the trajectory from pixels, but it keeps almost every intermediate a modular stack would have: depth, segmentation, 3D boxes, occupancy, agent forecasts, traffic-light state. Those are trained outputs, not internal activations, so you can still look at them. AutoE2E's v1.0 keeps far less: a bird's-eye feature grid and an auxiliary segmentation head used only in training, then a GRU that writes acceleration and curvature.

Eight cameras into one grid

METEOR's interface is spelled out in its model card and in meteor_v157.param.yaml. The inputs are eight RGB images at 768 by 432 from TIER IV's Co-MLOps data recording rig (wide, left and right, front and back, plus a narrow telephoto each way), each camera's intrinsics K and its extrinsic transform T_cam_ego, and the current speed v0. That is all. No HD map, no localization, no LiDAR unless you opt in.

Every image goes through the same ResNet-34 with a feature pyramid, producing a 160-channel feature map at a quarter of the input resolution: 108 by 192 per camera. Several heads read that map directly: a 21-class segmentation, a 10-class 2D detector, and a depth decoder that outputs, for each pixel, a probability over 64 depth bins out to 80 m.

The hard part of any camera-only driving model is the next step: putting features from eight perspective images onto one top-down grid. METEOR calls its answer a depth-gated IPM, and docs/ARCHITECTURE.md describes it in four lines. Here it is slower.

Lay a flat grid on the ground around the car: METEOR's output grid is 800 rows by 500 columns at 0.2 m, so 80 m ahead, 80 m behind and 50 m to each side. In the released architecture (DepthSegIPMNetV52) the lift itself runs on a coarser 400 by 250 grid at 0.4 m, four times fewer cells to sample, and the result is bilinearly upsampled to 0.2 m. Each cell has a 3D position $\mathbf{x} = (x, y, 0)$ in the car's frame. For camera $c$, project it into the image:

where $\mathbf{u}_c$ is the pixel the cell lands on and $r_c$ is the distance from the camera centre $\mathbf{o}_c$ to the cell. Sample the context features $f_c(\mathbf{u}_c)$ there (96 channels, bilinear). Then read the depth head's probability at range $r_c$ (interpolated between the two nearest bins), and use it as a gate:

The intuition is occlusion. The ray through pixel $\mathbf{u}_c$ passes over many ground cells before it hits something. Plain inverse perspective mapping paints the pixel's features onto all of them, which is why IPM images smear a parked car into a long streak across the road behind it. With the gate, a cell gets the pixel's features only where the network believes that pixel's surface actually is at that cell's distance. My reading of the 0.05 floor is that it keeps a gradient flowing where the depth head is confidently wrong. The weighted average across cameras settles which camera wins a cell that several of them see.

There are no learned parameters in this step, which is why the architecture table lists it at 0.02M with the context projection. That has a consequence the repository spells out in docs/REPRODUCE.md: every sample point has $z = 0$. The lift assumes a flat world. Depth never moves where a feature lands; it only decides which camera's feature counts. By that construction, a steep ramp or a raised kerb is projected as if it were painted on the road.

expand

The fused BEV in the diagram, a memory of three past frames warped by ego motion, is baked out of the release. More on why below.

The planner is a small head on a big perception model

Once the bird's-eye grid exists, every BEV task is a convolutional head on it. Lanes are a 9-class segmentation at full 0.2 m resolution. 3D boxes are CenterPoint-style heatmaps and regressions at 0.4 m. Occupancy is a crop of the grid to 40 m either way, lifted to 16 height bins. The E2E planner is one more head.

From bevlane/model.py: a pyramid of strided convolutions takes the 96-channel BEV down by 32x, from 800 by 500 to 25 by 16, and averages it to one vector; the speed is concatenated; an MLP emits 42 numbers. Later rounds added two branches whose last layers start at zero, so they began as no-ops: an attention pool with three learned queries over the BEV, and a small path that reads the lane-segmentation and box heatmaps instead of raw features, added because the global mean of raw BEV features collapsed under INT8. Those are three candidate paths of six waypoints each at 0.5 s spacing (a 3 s horizon), three confidences, and steering, acceleration and a brake logit. The released ONNX exports exactly that as its ego tensor, [1, 42].

Three hypotheses instead of one is the standard fix for averaging. At a fork, half the training data turns left and half goes straight; a single regression head learns to drive into the median. With $K = 3$ heads trained winner-take-all, one can learn the turn and another the straight. A later round bound the three modes to an optional driving command (straight, left, right) by adding a fixed logit to the commanded mode, so a route planner can pick between them. The released export drops that input (--no-intent), so at deployment the network picks its own mode.

expand

The part a modular-stack engineer will like is bevlane/guardrail.py. After the network picks a path, a deterministic checker with no learned weights tests it against the network's own other outputs: does the path, stepped at 0.5 s, intersect any detected agent moved along its own forecast to the same time; is there a red light with a stop line in the corridor; is the curvature feasible at this speed under a bicycle model; do the path cells lie on drivable classes in the predicted lane map. If any check fails, it vetoes the plan and substitutes a minimum-risk stop in lane. The docstring calls the arrangement doer/checker, and points out that the geometry heads read the single-frame BEV while the planner read the fused one, so the checker had a partly independent view.

This is where METEOR's choice to keep the intermediate outputs pays. A pure end-to-end model has nothing to check its plan against. METEOR has twelve outputs that can disagree with each other.

Now look at where the weights are. I read the released checkpoint, meteor_v157.pt, without running it: a PyTorch checkpoint is a zip file, its central directory lists every tensor storage, and its data.pkl can be walked with an unpickler whose class loader returns stubs, so no code from the file ever executes. Learnable tensors sum to 54,060,013, all float32 (BatchNorm running statistics, another 56,688 values, excluded). By module, the ResNet-34 trunk is 21.28M and the refiner 5.03M. The E2E planner, counting its convolution stem, MLP, attention pool, heatmap path and command pieces, is 4,521,408, or 8.4% of the network; adding its small residual refiner (0.11M) makes it 8.6%. The other nine tenths exist to see.

The compute is distributed differently again. The repository's measured table for an earlier version (v26) shows the backbone with more than half of the parameters and under a sixth of the GFLOPs, while the depth decoder and the BEV lane head, about a ninth of the weights, take nearly two thirds of the compute. They run at high resolution: the depth decoder over eight cameras at 108 by 192, the lane head over all 400,000 BEV cells. The widget toggles between that table and my count of v157.

FLOPs per frame times frames per second is the sustained compute a chip has to deliver. docs/REPRODUCE.md measures the v52 graph at 3,405 GFLOP per 8-camera frame; at 15 Hz that is 51 TFLOP/s, at 10 Hz 34. That is an upper bound for v157: its checkpoint carries the half-width depth head (0.92M parameters), which the same document says removes 785 GFLOP, so by subtraction a v157 frame is nearer 2,620.

That table matters for the "runs on any SoC" claim. Parameter count tells you how much memory the weights take: 54 MB at int8, 108 MB at fp16. It says little about whether a chip can run the model at 10 or 15 frames a second, which depends on FLOPs, on activation memory traffic at 800 by 500, and above all on whether the chip's compiler accepts the operators. METEOR's lift uses grid_sample and gather. On the Jetson AGX Orin it runs as a custom TensorRT plugin. That is the real portability problem, and the repository treats it as one (more below).

Where "no labels" comes from

Twelve heads need twelve kinds of ground truth, and nobody drew any of it. docs/DATA_PIPELINE.md describes what TIER IV calls the autolabel factory, built on its CoMET platform: a resumable pipeline (the README counts 17 stages) from raw recordings in the t4dataset format (a nuScenes-like layout with LiDAR sweeps, ego poses, camera frames, and 2D panoptic masks and 3D boxes produced by CoMET's own autolabelers).

• The trajectory label is free. The planner's targets are the car's own future poses at +0.5 s to +3.0 s, transformed into the current ego frame. Speed and acceleration are smoothed finite differences of the poses. Steering comes from a bicycle model, $\delta = \arctan(L \cdot \dot\psi / v)$, masked below 0.5 m/s. Brake is acceleration below −0.5 m/s². No CAN bus needed; the docs say so in the heading.
• BEV lanes come from projecting every LiDAR point into all eight cameras, reading the panoptic class at that pixel, accumulating class hits on a 0.1 m grid in the world frame across the whole drive, then vectorising, cleaning and re-rendering a per-frame crop. Thin classes like lane lines are only sampled within 15 m, because pixel-to-ground error grows with range.
• Depth is LiDAR splatted into each image and interpolated over the road segment.
• 3D boxes are kept only when a LiDAR box, projected into a camera, overlaps a same-class 2D box. Occluded objects never enter the BEV ground truth.
• Adverse weather is synthetic: NVIDIA Cosmos Transfer re-renders real scenes as night, rain, snow or reflective wet road while the labels from the original drive stay valid.

So "no labels" means no human annotation for this model's training, which is true as far as the repository shows. It does not mean self-supervised. The supervision is distilled from LiDAR, accumulated across a whole drive with hindsight the car does not have at inference time, plus the outputs of CoMET's own detectors and segmenters. How those autolabelers were themselves trained is outside this repository, and I could not check it. The method is also only as available as the data: training uses 953 GB of converted recordings across 12,595 scene directories on 8 data-center GPUs, and none of that is public. The published dataset is six anonymised demo scenes with no ground truth.

That changes how to read "anyone can develop it once the spec is set." Anyone with a TIER IV-style sensor rig, a LiDAR-equipped fleet, a CoMET-class autolabel stack and a GPU server can. The README says as much in a commit from day one: "Have Co-MLOps data? You can build this."

What "designed by agentic AI" means in the repository

The README is direct: every line in the repository, including models, label extractors, the trainer, the Python and C++ TensorRT runtimes, the CUDA plugin, the renderers, docs and slide decks, was written by Claude operating autonomously, with humans setting goals and reviewing results. The badge names Claude Fable 5.

The git history is consistent with that and can't prove it. Of 165 commits across all branches, 142 carry a Co-Authored-By: Claude trailer (118 of them Fable 5, the rest Opus versions). All are committed under one engineer's accounts; the citation names Dan Umeda as author. A trailer records that an agent took part, not that no human typed a line, so the "zero human-written code" badge stays the project's claim.

What the documents show more clearly is the loop the agent ran:

1. Rounds of about 24 hours. Each round refreshes the scene list with newly converted recordings, initialises from the previous best checkpoint and retrains every task. docs/TRAINING.md logs sixteen rounds and six new heads in the first five days.
2. One lever per round, judged automatically. From a workstation, the agents drive an 8-GPU training server and a Jetson AGX Orin. The judge reads validation metrics, a chained-rollout evaluation and held-out adverse scenes; the Orin rebuilds, calibrates and benchmarks every candidate. Adoption is decided by accuracy per millisecond.
3. Supervision that watches the supervision. A watchdog relaunches dead or hung runs; a cron supervisor catches subtler failures, like the round that ran for 34,280 steps while 23,533 of them were discarded as non-finite and validation kept printing plausible numbers.
4. No automatic recipe changes. An earlier watchdog dropped a loss after three failed starts. That behaviour was removed: the docs argue that a round silently training a different objective from the one on record is worse than a round that stays down until the bug is fixed.

The worst one is in the release. The architecture has a temporal memory of three past BEV frames. In September the team found that a dataset cache bug had fed zero history to every E2E training round since July. Feeding real history to those weights made trajectories 33 to 38% worse, and two rounds of retraining did not recover it. So the deployed engine is single-frame: the history path is baked out at export, which also took latency on the Orin from 107 to 83 ms. A history-aware line trained from scratch is listed as open work.

I count that disclosure in the project's favour. It is also the best evidence I have about agent-run ML. The judge checked accuracy, latency and robustness every round for two months and never noticed that an input it was supposed to be learning from was all zeros. An automated loop optimises what it measures, and nobody had written "is the history tensor non-zero" as a test until a human-scale question exposed it. If you run agent loops for training (as in auto-gpu-kernel, where the correctness gate held), the lesson is that the gates you write are the only things the loop will never get wrong.

The checkpoint's own saved arguments add one more detail. v157 was trained from v151's best planner checkpoint with freeze_ego: True and rl_w: 0.0: the final round applied 2:4 structured sparsity to the convolutional trunk with the planner frozen and the GRPO-style reinforcement stage that exists in bevlane/e2e_reward.py switched off. Whatever reinforcement shaped the planner happened in an earlier round, if at all.

Sixteen million, or fifty-four

The post says 16 million parameters. The released checkpoint has 54,060,013. The model card, the README, the parameter file and the architecture figure all say 54M. I searched the code, docs and commit messages on every branch for 16M, 16 million and 1600万 and found none. The video in the post is the eight-camera demo renderer, with the occupancy panel and guardrail readout of the METEOR demo; it is not the one-camera Nano, though from the footage alone I cannot rule out an eight-camera METEOR2 checkpoint.

There are smaller members of the family, and one of them is my best guess at the source of the number, labelled as a guess. METEOR2 swaps ResNet-34 for ResNet-18, whose trunk the repository measures at 11.18M against 21.28M, and replaces the plugin-based lift with a learned low-rank mapping built only from standard ONNX operators. METEOR2 Nano goes further: one front camera at 720 by 1280, ResNet-18, and a polar lift in which strided convolutions collapse each image column into range bins, so the BEV lives on an azimuth by range grid and the whole graph is about 220 Conv, Pool and Concat nodes. The README reports that version at 67.7 ms (14.8 FPS) as a single compiled graph on an unnamed low-power edge accelerator. Either a ResNet-18 METEOR2 with slimmer heads or the one-camera Nano could plausibly land near 16M. Neither METEOR2 nor Nano weights are released ("coming soon"), so I cannot count them.

The released model's numbers are concrete. On a Jetson AGX Orin at INT8 with CUDA graphs, the v157 engine with a 2:4 sparse trunk runs in 67.4 ms median, 14.8 FPS; the dense v151 engine runs in 75.4 ms; adding the LiDAR input costs half a millisecond. The C++ renderer adds about 30 ms of CPU work in parallel. The README itemises how 107 ms became 67: baking out the unused history (24 ms), pinned zero-copy input (5), the depth head at half width (1), a validity check that the compiler executed on a zero tensor (3), and sparsity (8). It also notes that an in-graph check for "is the LiDAR input non-zero" cost 14.8 ms by itself.

"Runs on any SoC" is the claim the family is built to approach, by rewriting the camera-to-BEV lift for each compiler's operator set. For the released METEOR it means: an AGX Orin with a custom CUDA plugin, or any NVIDIA GPU through a plugin-free fp16 engine (about 30 ms on a data-center card, per the card).

AutoE2E, the model it was quoted against

AutoE2E is developed in the open by the Autoware Foundation's Robotaxi working group, and its design is the other end of the line.

expand

The v1.0 release is the reactive branch alone. Six surround cameras at 512 by 512, plus an optional 1024 by 1024 front view, go into a BEVFormer V2 encoder with a ResNet-50 backbone that attends over the current frame and seven past frames at 0.5 s intervals. A 14-channel HD-map raster and a 2-channel route raster are fused into the BEV with deformable cross-attention. A GRU planner reads that, plus 64 steps of ego-motion history, and writes acceleration and curvature for 64 steps at 10 Hz: a 6.4 s horizon, twice METEOR's.

The Hub config.json gives the parameter count as 79,906,522, and the checkpoint agrees: its model tensors sum to 80,626,522, of which 720,000 are a fixed reference_points_3d buffer. The config also says the camera BEV encoder is frozen. The optimizer state saved in the released nuplan-epoch-5 checkpoint holds Adam moments for 1,448,582 parameters. So in that training stage, about 1.8% of the 79.9M were trained: the planner, the map and route encoders and their fusion. The rest is BEVFormer V2 as initialised from its official weights, which were trained on nuScenes. Two consequences follow. AutoE2E's "80M" is mostly someone else's perception model. And its checkpoints are licensed CC BY-NC-SA 4.0, non-commercial, because of nuPlan, KITScenes and those nuScenes-trained weights. METEOR's weights are Apache-2.0.

The AutoE2E README has a benchmark on KITScenes-Multimodal validation, a public driving dataset from Karlsruhe, and it includes METEOR. Every row is open-loop: the model predicts 3 s from logged frames and is compared with the human's path.

METEOR's README has its own table on the same 202 KITScenes keyframes, for METEOR2 fine-tuned on KITScenes train:

Read side by side, each project benchmarks the other at a disadvantage, without anyone doing anything wrong. AutoE2E could only test what METEOR released: v157, trained on a Japanese 8-camera rig, run zero-shot on a German 6-camera rig, with the missing narrow camera synthesised by cropping the front view (run_meteor_kitscenes.py notes that a black slot instead made METEOR under-predict progress by about 10%). Its 1.240 m is a cross-domain number. METEOR's 0.542 m is a fine-tuned model whose weights are research-only and not released, with a route built from the recorded drive, which the README calls a navigation oracle. Different evaluation code computes both tables.

What I would take from them: with comparable inputs to AutoE2E's fine-tune (cameras, ego history, HD map, route), METEOR2's fine-tune reports a lower 3 s error, 0.542 against 0.744. That compares two codebases' metric scripts, and METEOR2's size is unpublished, so treat the gap as suggestive. And neither table says anything about whether either model can drive.

What "open-source FSD" would need

The repository itself lists most of what stands between METEOR and a car:

• No closed-loop evaluation. bevlane/closed_loop_eval.py is the judge's primary metric, and its docstring calls it "quasi-closed-loop": follow the model's own selected plan for one 0.4 s step, accumulate the predicted displacements over eight steps, compare the chain with the logged chain. It explains why: "True closed loop needs re-rendered images from the displaced pose, which log replay cannot produce." The chain shows how errors compound, but a model drifting toward the kerb never sees the kerb get closer.
• A thin margin over doing nothing. The training docs decompose the validation error every epoch, including a constant-velocity, keep-going-straight baseline. On one round, they report, the model beat it by less than a quarter, and about a quarter of the error was the planner picking the wrong one of its three hypotheses.
• Single-frame. No way to remember a pedestrian who stepped behind a van.
• One domain. Japanese roads plus a US subset, one traffic-light convention, one sensor rig. Other rigs need retraining.
• No integration. There is no Autoware ROS 2 node yet; METEOR ships its own runtimes.
• The publisher's own line. The model card's limitations section: "Not a certified driving system; it must not control a vehicle on public roads."

To that list I would add what any safety engineer would ask for first: an operational design domain (which roads, speeds, weather, times of day), a hazard analysis, and a safety case arguing that the system with its guardrail and a fallback meets a target in that domain. AutoE2E at least has the start of one, a written review of Mobileye's Responsibility-Sensitive Safety framework in Safety/RSS/ as the basis for its safety argument; the review says itself that it is not a certification artifact. METEOR's guardrail is the right architectural instinct. It is also four rules checking a model against itself.

Tesla's FSD is a product: a fleet, validation that closes the loop on real roads, a supervising driver as the fallback, and a regulatory status. "Open-source FSD" borrows the name of that whole system for one component of it. METEOR is a strong open perception-and-planning reference model with permissive weights, a reproducible training stack for whoever holds the data, and a detailed record of an agent running an ML project since July. That is plenty. It does not need the comparison.

If you work on driving perception, the part I would read first is docs/REPRODUCE.md §7, "Measuring so the numbers mean something". Its account of how measuring SD-map and LiDAR inputs the wrong way produced "cost 19 % of mIoU" and "did nothing", both wrong in opposite directions, is useful whoever writes your code. For comparison with a larger, VLM-based approach to the same job, see Qwen-Drive-1.0, which leads AutoE2E's table; for a model that plans from imagined video rather than a bird's-eye grid, Praxis-1.