Bonsai 27B: Growing a 27B Model Down to Phone Size

Why this post

On July 14, 2026, PrismML released Bonsai 27B. It's the first time a 27B-class model has been compressed down to 3.9GB and actually run on an iPhone. This post covers three things. First, an intuitive read on how 1-bit quantization works. Second, how far the claim that it "runs on an iPhone" actually holds up against real-world measurements. Third, what changes from the perspective of a platform engineer who owns inference infrastructure.

TL;DR

Bonsai 27B is a multimodal model, retrained from scratch with 1-bit (±1) / ternary (±1, 0) weights on top of Qwen3.6 27B. 3.9GB / 5.9GB, Apache 2.0, 262K context.
Measured at roughly 11 tok/s on an iPhone 17 Pro Max. About 672 tokens per 1% of battery. Thermal throttling kicks in after a little over 5 minutes, though.
Math and coding stay close to the original, but tool calling and vision drop off. Not quite ready for production agentic systems.
Core takeaway: the center of gravity in inference is widening from "running GPU clusters" to "managing heterogeneous deployment."

How it works — turning dials into switches

Starting with the name actually makes the mechanism easier to follow. "Bonsai" (盆栽) describes the practice of growing a tree inside a small pot (盆), continually pruning its branches and roots so the tree keeps the shape and vitality of a full-size tree while staying small. The key point isn't shrinking a mature tree after the fact — it's training the tree, from the very start, to grow inside that small container. PrismML hasn't officially explained the name, but — as you'll see below — it lines up with what this model actually does: a 27B-class "tree" that wasn't grown large and then cut down, but was retrained end to end from the beginning to grow inside a phone-sized container.

A model's weights are like volume dials. Ordinarily, a model stores each dial as a precise value like 0.7362941, using 16 bits. At 27.8 billion of them, that's about 54GB — nowhere close to fitting a phone's per-app memory budget (roughly 6GB on a 12GB iPhone).

Bonsai's approach is to turn every dial into a switch. Each weight can only be +1 or -1 (1-bit). Every group of 128 weights shares a single FP16 scale factor to compensate for the precision loss, which works out to an effective 1.125 bits per weight.

The order of operations matters here. This isn't a fully-trained model that got crushed down afterward — the "only switches allowed" constraint was imposed from the start, and the entire pipeline (embeddings, attention, MLP, LM head) was retrained end to end under that constraint. It's the same idea as a painter who trained from day one to draw only with a single black pen, and who still produces good work in black and white. PrismML claims 90% (1-bit) / 95% (ternary) of full-precision performance retained, based on its own set of 15 benchmarks.

There's a speed bonus too. ±1 multiplication reduces to addition and subtraction, and memory bandwidth — the usual inference bottleneck — drops to roughly 1/14th. On the CUDA path, packed weights aren't unpacked back to FP16; a fused 1-bit GEMM kernel consumes them directly.

Real-world testing — the gap between "it runs" and "it's usable"

Here's what's been measured in the five days since launch.

About 11 tok/s on an iPhone 17 Pro Max. 87 tok/s on an M5 Max, 163 tok/s on an RTX 5090.
Roughly 672 tokens per 1% of battery, about 67,000 tokens on a full charge. Thermal throttling begins after a little over 5 minutes, though.
Runtime paths: an mlx-swift fork (iOS/iPadOS), MLX (Apple Silicon), a llama.cpp fork's GGUF build (CUDA), and a Together AI-hosted developer preview API. Everyday users can run it fully offline through the Locally AI app.

There are parts worth filtering out, too. PrismML's multimodal agentic demo ran in a "cached, preloaded image context" demo mode. At 11 tok/s, it's below the roughly 15–20 tok/s threshold where a conversation feels natural — so it fits background processing, document analysis, and offline batch work better than real-time chat. And the quality retention isn't even across the board — reports suggest math and coding stay close to the original, while tool calling and vision drop off.

Implications — what actually changes

The boundary around cloud-only capability is eroding.

The assumption behind "small models run locally, big models run in the cloud" is getting shakier. Once multimodality, a 262K context window, and tool calling all fit on a phone, the answer to "should this task go to the cloud?" is going to be "no" a lot more often.

Inference economics are being reshaped.

On-device inference has a marginal cost of essentially zero. Cheap, high-frequency workloads staying local while frontier-level reasoning goes to the cloud — an "easy stuff local, hard stuff cloud" hybrid routing setup — looks likely to become the standard architecture.

The default posture on privacy and regulatory compliance shifts.

When data never leaves the device, adoption barriers come down in domains like healthcare, finance, and law, where data can't easily leave in the first place. Reports that Apple is evaluating PrismML's technology fit this same context.

It signals a direction for hardware.

If NPUs that natively support ±1 operations show up, power efficiency improves dramatically. If low-bit becomes the norm, it could reshape how the next generation of mobile and edge chips gets designed.

The open ecosystem is on the offensive.

The pattern here is: take an open model (Qwen), compress it, and re-release it under Apache 2.0. That accelerates a broader shift where closed APIs' moat moves away from raw model capability and toward infrastructure, reliability, and orchestration. The next target, reportedly, is compressing Gemma.

The platform engineer's view — new problems on the table

The center of gravity for inference infrastructure is widening from "running a massive GPU cluster" to "managing heterogeneous deployment." Concretely, that means new problems like these:

A deployment matrix per quantization variant. Deciding and managing which variant (FP16, ternary, 1-bit) the same model gets deployed as, across cloud GPU, on-prem, and edge.
Monitoring for quality degradation. Even at a 90% benchmark average, specific tasks (tool calling, long reasoning chains) can silently collapse. Per-task quality gates become necessary.
Local-to-cloud fallback routing. Designing routing that falls back to the cloud when local inference fails or drops below a quality threshold.
Verifying the model artifact supply chain. Tracking where a given quantized build actually came from — structurally, this resembles SBOM/signature verification. The same way Cosign verifies container images, demand will grow for verifying the provenance of model weights.
Orchestrating edge workloads. KubeEdge-style edge inference orchestration is turning into an actual operational requirement.

Caveats

The 90–95% performance retention figure is based on PrismML's own benchmarks. Independent verification is still pending.
Low-bit models often look fine on average scores while remaining fragile on long reasoning chains, rare knowledge, and tool calling. Production adoption should wait on community verification.
Sustained on-device performance (throttling) is a real bottleneck for agentic workloads.