# Inkling: an open-weights multimodal MoE built to be adapted > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.satyajit.si/articles/inkling > date: 2026-07-15 > tags: mixture-of-experts, multimodal, reinforcement-learning, attention Most model launches lead with a leaderboard. Thinking Machines Lab's **Inkling** does the opposite: the announcement states plainly that it is **"not the strongest overall model,"** and is instead **"designed for broad adaptation through fine-tuning."** That framing is the right lens for everything below. Inkling is an **open-weights**, multimodal **[mixture-of-experts](/articles/mixture-of-experts-from-scratch)** foundation model — **975B total parameters, 41B active** — with text, image and audio in one stack and up to a **1M-token** context. A smaller companion, **Inkling-Small (276B total / 12B active)**, ships in preview. The pitch is a customizable *base*, not a frontier trophy. What makes it worth a close read is the mechanics: an attention design tuned for long context, an encoder-free multimodal path, a reinforcement-learning run whose reward scaled *log-linearly* while the model's reasoning got *shorter* on its own, and a knob that lets you dial how many tokens the model spends per query. Let's take them in turn — and keep the honest caveats in view throughout. ## The backbone: sparse experts, hybrid attention Inkling is a **66-layer** decoder-only transformer. Two forms of sparsity run through it. In the feed-forward path, every layer is a mixture-of-experts: **256 routed experts plus 2 shared experts**, with **6 routed experts active per token**. A **sigmoid-based router** with an **auxiliary-loss-free load-balancing** bias decides which six fire — the same "drop the aux loss, use a bias term" trick that has become standard for keeping expert utilization even without a loss that fights the main objective. The 2 shared experts are always on, giving every token a common backbone of computation. Net effect: only **41B of the 975B** parameters do work on any given token. The attention path is a **hybrid**: of the 66 layers, **55 are sliding-window (512-token) local and 11 are global** — an interleaved **5:1 ratio** — with **64 query heads** tied to **8 KV heads**, over a **6144-dim** residual stream. Five cheap local layers pass for every one exact global layer — the same local/global bargain that makes long-context serving affordable in [MiniMax's sparse attention](/articles/minimax-sparse-attention) and [MiMo-V2-Flash](/articles/mimo-v2-flash). Then a cluster of small but telling choices — the kind you only catch by reading the config, not the launch post: - **Relative position bias**, not [RoPE](/articles/how-llm-inference-works) (`d_rel=16`, `rel_extent=1024`) — a learned bias on relative distance, chosen for cleaner extrapolation past the trained length. - **Short depthwise convolutions** (kernel size 4) in *several* places — after the key and value projections and on the residual branches — a cheap way to blend a little local context into each token before attention even runs. Convs-inside-a-transformer is a recurring "free lunch" for stability. - An easy-to-miss one: a **separate RMSNorm on the token embeddings**, applied *before* the usual per-block RMSNorms (`use_embed_norm=true`). The residual stream is normalized at *entry*, not only inside each layer — extra insurance on embedding scale that most decoder-only stacks skip. None of these are headline features; together they read as the fingerprint of a team tuning the backbone for stable long-context training rather than chasing a benchmark. Scrub the stack to see both sparsities at once — which layers are global, and which experts a token lights up: ## Encoder-free multimodal The multimodal design is deliberately minimal: **no separate vision or audio encoder**. Instead every modality is turned into tokens the transformer reads directly. **Audio** becomes **discrete dMel spectrogram** tokens; **images** are cut into **40×40-pixel patches** and lifted by a small **four-layer hMLP** patch encoder; all modalities land in the **shared hidden space** and flow through the same experts and attention. There's no bolted-on CLIP-style tower whose representation you have to align — the model learns text, image and audio in one backbone. That is part of why it's pitched as an adaptation base: fine-tuning touches one stack, not a federation of encoders. ## Controllable effort — the signature move Inkling can vary how much it "thinks." The **system message plus a per-token cost** let you trade accuracy for token spend: turn effort down and it answers tersely; turn it up and it reasons at length, approaching its ceiling. The headline result is on **Terminal-Bench-2.1**, where the lab reports Inkling reaching **Nemotron-3-Ultra-equivalent** accuracy at **roughly one-third the generated tokens**. Drag the effort knob and read the tie line — the same score sits about **3× further right** on the reference curve: The efficiency framing matters more than any single point on the curve. A model that lets the *caller* choose the accuracy/latency trade-off, per request, is a different product from one with a fixed thinking budget — especially for the fine-tuning-and-deploy audience Inkling targets, who care about tokens-per-task cost at scale. (The curve shape above is illustrative; the ~63.8% Terminal-Bench-2.1 plateau and the ~1/3-token match are the real, vendor-reported anchors.) Here is the sweep as Thinking Machines plot it, which is the measurement the interactive above is drawn from:
## The RL story: log-linear reward, self-shortening reasoning Post-training leaned on **large-scale asynchronous [reinforcement learning](/articles/ring-zero-trillion-scale-rl)** — **over 30 million rollouts**. Two findings stand out. First, the **aggregate held-out eval reward rose log-linearly** across those rollouts, climbing from **0.264** at the SFT-initialised checkpoint to **0.356** at release — a straight line on a log-rollouts axis, i.e. more RL compute kept paying off predictably rather than saturating. Second, and more surprising: with **no brevity objective** in the reward, the model's **chain-of-thought became more concise on its own**, "dropping grammatical overhead while remaining comprehensible." Reasoning compression emerged as a side effect of optimizing for correct answers. Drag the marker to watch reward climb as thought-length falls:
This connects back to controllable effort: a model whose reasoning is naturally terser is cheaper to run at any accuracy target, and the effort knob then lets you push that further. ## Training and release Pretraining ran on **45 trillion tokens** of mixed text, image, audio and video, optimized with **[Muon](/articles/muon-optimizer) for the large matrix weights and Adam for everything else** (weight decay coupled to the squared learning rate), on **NVIDIA GB300 NVL72** systems. Alongside the standard weights, Thinking Machines released **[NVFP4](/articles/nemotron-nvfp4)** weights for Blackwell — the same 4-bit format NVIDIA used to train Nemotron. The release is genuinely open: **weights on Hugging Face** (both standard and NVFP4), an **API on Tinker plus Together, Fireworks, Modal, Databricks and Baseten**, day-one **vLLM / SGLang / llama.cpp** integration, and a public **Playground**. ## Results — read them as vendor-reported
Here are the headline numbers from Thinking Machines' own suite (at high effort). Reasoning first: And the agentic / coding side, where the effort story is most relevant: Multimodal and safety round it out: **VoiceBench 91.4%**, **MMAU 77.2%**, **MMMU-Pro 73.5%**, **Global-MMLU-Lite 88.7%**; on safety, **FORTRESS Benign 95.9% / Adversarial 78.0%** and **StrongREJECT 98.6%**. Inkling-Small tracks the big model closely on several evals (HLE text-only **31.6%**, HLE with tools **47.8%**). **Scope the numbers.** Every score here is **vendor-reported on Thinking Machines' own evaluation suite**, and the comparison set (GPT-5.6 Sol, Claude Fable 5, GLM 5.2, Nemotron-3-Ultra, and others) is **provider-selected** — so treat this as a self-report, not a neutral head-to-head. The "~1/3 the tokens," the log-linear RL scaling (0.264 → 0.356), and the emergent reasoning-compression claim are all **their measurements on their evals**: real and interesting, but not independently verified. And this is a **company blog and model card, not a peer-reviewed paper** — there is no external methodology to audit. The genuine mitigant is that it's an **open-weights** release, so the architecture and the claims become independently checkable over time in a way a closed model's never are. ## The take Inkling's most refreshing feature is its honesty about what it is. Thinking Machines did not build the model to top a leaderboard; they built a broad, open, multimodal base and tuned the *ergonomics of adapting and running it* — controllable effort so callers own the accuracy/cost trade-off, a 5:1 local/global attention hybrid and relative positions so 1M-token context stays affordable, an encoder-free multimodal path so fine-tuning touches one stack, and NVFP4 weights so it deploys cheaply on Blackwell. The two research results worth remembering are the **log-linear RL reward** (evidence that the post-training recipe kept scaling) and the **emergent reasoning compression** (shorter chains-of-thought with no brevity reward) — both their own measurements, both the kind of thing open weights will let others probe. Judge it not as "is this the best model" — the lab already answered no — but as a base you can take, fine-tune, and serve. On that axis, an open 975B/41B MoE with these ergonomics is a substantial thing to hand the community. --- *Built on Thinking Machines Lab's [Inkling announcement](https://thinkingmachines.ai/news/introducing-inkling/) and [model card](https://thinkingmachines.ai/model-card/inkling/), plus the [Hugging Face release](https://huggingface.co/thinkingmachines/inkling). All benchmark and scaling figures are vendor-reported; the interactive diagrams are illustrations of the mechanism, with real endpoints noted inline. The three static figures are the announcement post's own charts, reproduced as published — the Hugging Face repository ships no figures of its own, and the model card page carries none either. Related reading: [mixture-of-experts from scratch](/articles/mixture-of-experts-from-scratch), [NVFP4 training](/articles/nemotron-nvfp4), [MiniMax sparse attention](/articles/minimax-sparse-attention), [MiMo-V2-Flash](/articles/mimo-v2-flash), [trillion-scale RL](/articles/ring-zero-trillion-scale-rl), [the Muon optimizer](/articles/muon-optimizer), and [how LLM inference works](/articles/how-llm-inference-works).*