Agentic Inference Optimization · Kimi K3 · AMD MI355X
Kimi K3 on AMD MI355X: +40% Agentic Throughput, +86% Peak vs. NVIDIA B300
Is CUDA’s Performance Moat Shrinking in Specific Workloads?
We optimized Kimi K3 agentic inference end to end on a single 8×AMD MI355X node: kernels, KV cache, communication and speculative decoding. On InferenceX, which replays real coding-agent sessions, it serves 40% more tokens per GPU than NVIDIA B300's best public result, and 86% more with speculative decoding.[1]
Our finding: for a given model and workload, performance is increasingly a software problem, not a hardware limit. AI is making these optimizations far more accessible, and that is narrowing CUDA's long-standing moat.
Inference optimization now moves week by week. Our results suggest that alternative compute is no longer a future possibility but a viable, increasingly competitive choice for frontier AI inference, especially as depending on a single GPU vendor grows more costly.
- +40% without speculative decoding. 17,551 tok/s/GPU at c80 against B300's best public point, 12,566 at c70.
- +86% with speculative decoding. 23,330 tok/s/GPU at c80, at the benchmark's fixed "golden" acceptance rate. On natural-text traffic, speculation adds 14–28%.
- Quality held. GSM8K 96.74 against a 96.57 baseline, 100% needle recall. Pass Kimi Verifier Benchmark.
Agentic Inference · Long input with high cache rate
* Lorenz Research agentic inference stack. With deep optimized speculative decoding: DSpark+ ReplaySSM at the benchmark's fixed "golden" acceptance rate. Without speculative decoding: 17,551 at c80 vs. 12,566 (+40%); like for like at c48, +37% (15,681 vs. 11,428). B300 data: SemiAnalysis InferenceX public run published 2026-08-16. Its sweep peaks at c70, 12,566 tok/s/GPU, its highest tested concurrency; our stack still gains throughput at c80. c = concurrent agents; GPU-hour prices from InferenceX.
Throughput and cost against interactivity
Output throughput, tok/s per GPU
Tokens per $1
Hover a point or a line for its numbers. Change the GPU-hour prices above and the right chart redraws.
Tokens per dollar
On InferenceX benchmark.
| Configuration | tok/s/GPU | $ / GPU-hour | M tokens per $ | $ / M tokens | vs B300 |
|---|
Kimi Verifier Benchmark
Speed only counts if quality holds. Evaluated with Kimi Vendor Verifier[GitHub]
| Benchmark | Ours (Lorenz Research) | Other vendors |
|---|---|---|
| GSM8K (1,319) | 96.74 baseline: 96.57 | — |
| Needle recall, ~108K tokens | 100% | — |
| OCRBench | 0.884 | 0.878–0.897 |
| MMMU-Pro Vision | 0.815 | 0.804–0.820 |
| BEAM 1M | 0.2935 | 0.291–0.322 |
| DeepSWE | 0.655 | 0.658–0.695 |
Ours: the Lorenz Research speculative decoding setting, with deeply optimized kernels for DSpark+ReplaySSM.
Key technical implementations
One 8×MI355X node running vLLM on ROCm with our own patches. Here are some of the key changes we made (not all of it).
Exact stateful speculation. One recurrent KDA state per request, rolled back by replay instead of copied for every draft token. Output is bit-identical, and it runs with the host-DRAM KV tier.
A KV hierarchy built for agentic reuse. Prefix-cache entries inside a block, so forked sessions still hit; projected K/V cached as well; host DRAM as a second KV tier; KV capacity sized from the measured VRAM floor. Long shared prefixes stop being recomputed.
Attention shaped to the model and the chip. Decode and verify kernels written for K3’s 96-head MLA and the MI355X chiplet (XCD) layout, reading KV from HBM once.
Communication off the generic collective path. In-node traffic goes over IPC and the SDMA copy engines instead of a general collective library, fused with the residual and MoE sums next to it.
No work done eight times. Projections, the router and AttnRes, which every GPU used to repeat, are row-sharded and computed once.
Lower host overhead. Batch metadata built once per step, CUDA graphs that can break mid-step at laddered sizes, batch-sharded sampling, and the shared expert on a side stream.
Per-shape GEMM selection. Narrow GEMMs moved to dedicated kernels, with the best solution picked for each shape.
Every change that touches numerics passes bitwise, GSM8K and needle checks before it counts as a speedup.
What the speculative stack adds at c80
Each row adds items to the one above it, measured on AgentX c80 in the FAST window. One step at a time, B2B and the last row's items added up to +5.5%; stacked, they gave +6.6% over the push row, so nothing cancelled out.
Acknowledgements
This work stands on the vLLM and AMD ROCm ecosystems. Here is what came from where.
- vLLM is our serving engine (ROCm nightly 2026-10-07). Several of our changes target the same bottlenecks as AMD's vLLM 10-19 update for this benchmark: cheaper MLA metadata building (our MD1/MD2; upstream #58381), FP8 all-gather of the DCP prefill context (#59070), FP8 MoE stage-2 outputs, and narrow decode GEMMs for the router (#61895).
- AITER and FlyDSL provide our FP8 prefill attention, the INT4 all-reduce we build on, MoE kernels and IPC primitives. Our fused all-reduce and DCP communication extend them.
- ATOM showed that DSpark with ReplaySSM is the right decode path for Kimi K3 on MI355X. Our ReplaySSM follows the design of vLLM's RecoverSSM, informed by ATOM's ReplaySSM, and is re-implemented on vLLM's ROCm path to be bit-identical with the existing KDA kernels.
- Models, evaluation and benchmark. The DSpark draft model is Inferact's; Kimi K3 and the Kimi Vendor Verifier are Moonshot AI's; the InferenceX benchmark, harness and public results are SemiAnalysis's.
References
- SemiAnalysis InferenceX, Kimi K3: B300 vs MI355X, agentic. Source of every NVIDIA B300 number (B300 vLLM 08-16 public run, AgentX, 3600 s window), the AMD vLLM 10-15 and 10-19 runs and the ATOM 10-15 run, and the GPU-hour prices ($2.26 per B300-hour, $1.50 per MI355X-hour).
- SemiAnalysis InferenceX, Kimi K3 inference results (interactivity view, agentic traces, FP8). Public points per concurrency, including B300's highest point at c70 (12,566 tok/s/GPU).
- Lorenz Research measurements on one 8×AMD Instinct MI355X node with the InferenceX AgentX harness (aiperf 0.12.0, agentx scenario), October 2026. All numbers labeled "ours".
- MoonshotAI, Kimi Vendor Verifier (GitHub). The evaluation suite behind the Kimi Verifier Benchmark table (OCRBench, MMMU-Pro Vision, BEAM 1M, DeepSWE) and the source of the other-vendor K3 ranges.
Measured on our node with the InferenceX AgentX harness (aiperf 0.12.0, agentx scenario). Official window: 3600 s, 10 warmups per lane, as InferenceX runs it. FAST: 1200 s, 1 warmup. Speculative AgentX numbers use the benchmark's golden acceptance length (3.36); natural-text results use real acceptance. Throughput is total tok/s per GPU including cached input; interactivity is p90 tok/s per user. Configurations, patch lists and run logs are available on request at research@lorenzresearch.ai.