A research team led by Song Ci at Yuncun New Energy (Tsinghua-affiliated) just published what they describe as the first token-level joule measurement of LLM inference energy. I read it. Here is the short version for anyone shipping LLM-powered products: the model you route to matters far more for your API bill than most teams assume — and “bigger” is usually the wrong default.
If you build on an AI API gateway, this report is essentially a price-efficiency map for your routing table.
What they measured
Eight models, real hardware (8× NVIDIA GPUs, vLLM 0.19.1 and SGLang, the Zeus energy library), three task types (text, code, multimodal), and batch sizes from 8 to 128. Energy is reported in joules per token — the unit that actually shows up in your provider’s cost.
The headline rows:
| Model | Text J/token | Code J/token | Multimodal J/token | Note |
|---|---|---|---|---|
| Gemma4-31B | 0.35 | 0.47 | 0.84 | Efficiency champion, stable across scenarios |
| gpt-oss-120B | 0.37 | — | — | MoE + FP8, efficiency meets throughput |
| MiniMax-M2.7 Reap | — | 0.86 | — | ~4300 tok/s, speed king for code |
| Llama4-Maverick | — | — | 0.86 | Best for vision |
| DeepSeek-V4-Flash | — | 1.92* | — | *after switching to SGLang: 5.7× throughput, 78% less energy |
| Qwen3.5-397B | 1.07 | 2.18 | — | ~3× Gemma’s cost, volatile |
Three rules fall straight out of the data:
- Batching is the free lunch. Energy drops monotonically as batch size goes 8→128; gpt-oss fell more than 5×. Tuning concurrency often beats swapping models.
- Same hardware, 3× cost spread between models. At bs=128 text, gpt-oss needs 0.37 J while Qwen3.5-397B needs 1.07 J. Pick the right model and you roughly double throughput per dollar.
- MoE + low-bit is the winning efficiency route. gpt-oss (sparse activation + FP8) posts the best energy across scenarios — better than models with fewer parameters.
Why this matters if you call models through an API
You don’t pay for joules directly. But your provider does, and it shows up in the price per token. A gateway that routes your traffic toward efficient open-weight models can pass real savings to you. The production principles the authors land on are the same ones a good gateway is built on:
- Scenario decides the model — stop defaulting every call to the largest frontier model.
- Config decides efficiency — batch size and parallel strategy move the number more than people expect.
- Framework decides the result — they swapped vLLM for SGLang on the same DeepSeek model and got 5.7× throughput. The serving stack matters as much as the weights.
What we do today
We already aggregate cost-efficient models like DeepSeek V4 (see the developer guide) and MiniMax, and we are evaluating adding the efficiency champions from this report — Gemma4-31B, gpt-oss-120B, Llama4-Maverick — to our routing pool. The point is not “use the biggest model.” It is “use the right model for the job, and let the gateway handle the routing and the batching.” If you want to start now, the 5-minute quickstart gets you on a cost-efficient model today.
A practical picking rule
- High-volume Agent traffic / price-sensitive → small efficient models
- Text throughput at scale → MoE models (low bit)
- Code generation → speed-optimized models (~4300 tok/s class)
- Multimodal / vision → vision-efficient models
- Complex, quality-critical tasks → keep a frontier model in the loop
The report is a useful reality check: model capability and deployment cost are different axes. If your bill is the constraint, route by efficiency first and reach for the heavyweights only when the task earns it. For a concrete cost walkthrough, our peak pricing guide shows how routing alone cuts spend without code changes, and the Claude vs OpenAI cost piece covers the frontier tier.
Start routing by efficiency. Put a cost-efficient model on your first request today.
Benchmark data: 《大模型系统级单位Token能耗测评》报告 (Yuncun New Energy / Tsinghua, 2026-07-29).