swesty.net

vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide

Mon, 20 Jul 2026 00:00:00 GMT

A companion, published after the field notes. I published the field-notes post, vLLM on the DGX Spark, the Hard Way, first: that’s the operational story, what broke, what we run in production, and the measured numbers. This post is the deep background behind it, the kernel, CUTLASS/FlashInfer, quantization, and backend-oracle detail I had to work through to understand why those field observations happen at all. Researching it was most of the actual work, and it seemed worth putting in one place for anyone hitting the same walls, so I’m releasing it as a companion. Read the field notes for what to do today; read this for why it works underneath. One caveat: this is a dated implementation audit (2026-06-02 against vLLM main), so treat the version-specific operational bits as a snapshot and defer to the field notes’ timeline for the current state.

Why this report exists. On 2026-06-01 the vLLM team published its DGX Spark deep-dive. We implemented its guidance on our production fleet the next day, and then ran it for seven weeks of real traffic. This report is what that produced: a commit-level audit of the implementation details the high-level recipe leaves out (SM121 arch targeting, backend oracle behavior, dense-vs-MoE split routing, FlashInfer cubin/JIT behavior, CUDA graph escape hatches), plus an honest scorecard of what worked, what didn’t survive production, and what we had to discover ourselves. Where the official blog and repository behavior diverge, the repository behavior is treated as authoritative for the audited commit. The body is a dated snapshot of that audit; the update timeline records what has changed between the audit and this post’s publication.

Scope. This is an all-in-one technical reference for running and understanding vLLM on the NVIDIA DGX Spark. It covers the hardware, why the platform has been hard for vLLM, the kernel libraries involved (CUTLASS, FlashInfer, Marlin, QuTLASS), how quantization (NVFP4 / MXFP4 / FP8) is wired up, how vLLM’s fused-MoE machinery selects backends, what --enforce-eager is and why it has been a crutch, the development history, and concrete guidance on prebuilt containers vs. source builds.

Provenance. Code citations are against the vLLM main checkout at commit 689e... (2026-06-02). Release-timing and the operational recommendations are reconciled with the official blog “vLLM on NVIDIA DGX Spark” (https://vllm.ai/blog/2026-06-01-vllm-dgx-spark). Where the repo and field experience diverge, the repo wins and the divergence is called out.

⏱ Freshness & maintenance: implementation audit dated 2026-06-02 (vLLM main @ 689e…). v0.22.0 is treated throughout as the first audited stable baseline containing the Spark fixes discussed, not as the latest release as of this post’s date (stable releases have continued since; see the update timeline). Re-audit backend selection and dependency pins against the exact image or tag you deploy. sm_121 support is a fast-moving target (vLLM stable cadence ≈ every 2–3 weeks), so claims carry inline date tags: [verified YYYY-MM-DD] = checked against the repo/blog on that date; [since vX.Y.Z] = release a fix first shipped in. Time-volatile sections to re-check every vLLM release: §4.4 (the b12x-is-opt-in exclusion, the single highest-churn fact here; it flips when flashinfer-ai/flashinfer#3134 lands upstream), §7.2–7.3 (backend auto-selection order), §9 (release→fix mapping), §10 (recommended image + dependency pins). The architectural material (§1, §3, §5–§6, §8) is structural and ages slowly. When you re-verify, bump the date tag even if nothing changed: a current date is the signal the claim still holds.

What’s changed since the audit

The body below is the 2026-06-02 audit snapshot. Between that audit and this post’s publication (body assembled 2026-07-20; this timeline last extended 2026-08-05), the following moved. Each entry is either verified in our own production fleet (details and receipts in the GB10 field notes) or checked against the public release record on the date shown.

2026-06-05 → 2026-07-05: MTP under real concurrency [fleet-verified]: in-checkpoint MTP validated at 86.8% acceptance, then hit CUDA error: an illegal memory access every few hours in production, only under concurrent load (upstream #41190/#42084).
2026-07-11/12: native FP4 confirmed working in production [fleet-verified]: a genuine W4A4 checkpoint (4-bit activations, lm_head unquantized) put dense and MoE layers on the FLASHINFER_CUTLASS native-FP4 path, faster than forced Marlin on our boxes, and it eliminated the MTP crash (8-way soak, 400/400, zero IMA). §6’s checkpoint-format emphasis aged well; any lingering “Marlin is mandatory on GB10” reading of §5 did not.
2026-07-14: vLLM v0.25.1 released [PyPI, checked 2026-07-20]: multiple stable releases now postdate the audited v0.22.0 baseline.
2026-07-19: official cu129 stable still ships no sm_121 cubins [fleet-verified]: smoke-testing the v0.25.1 stable candidates reproduced the report’s central lesson: wheel/cubin coverage, not version number, decides whether a path works. Our fleet runs digest-pinned nightlies for exactly this reason.
2026-07-27: v0.26.0 released; cu129 aarch64 stable now HAS sm_121 NVFP4 cubins [fleet scratch-tested 2026-08-05]: auto selects FLASHINFER_CUTLASS for both dense and MoE with no no kernel image is available, which breaks the 2026-07-19 lesson above (the 0.22.0 → 0.25.1 cu129 cubin drought is over). It still could not serve our unsloth W4A4 checkpoint, though: EngineCore init dies on a CUTLASS FP8 W8A8 GEMM (cutlass_gemm_caller.cuh:62, Error Internal), a path the same checkpoint clears on our cu13-line nightly. The lesson refines rather than breaks: the failure moved from a missing NVFP4 kernel image to an FP8 CUTLASS error, still wheel/CUDA-line-specific, not version-specific.
2026-07-29: #41190 (MTP illegal-memory-access) fixed on main [PR #48245/#49620]: root cause turned out to be a kernel-agnostic scheduler bug (async-decode output-placeholder underflow under KV-pressure preemption), not the drafter kernel. Not yet in a tagged stable; upstream marks it a probable “may fix.” Stable-wheel operators stay exposed until a nightly carries it.
2026-08-05: #42084 (prefix-caching + spec-decode OOB) appears resolved on a v0.26.1-dev nightly [fleet scratch-tested]: prefix caching and MTP now run together via an experimental Mamba/SSM cache-align mode (it aligns the attention block size to the linear-attention page size). Validated with a shared-prefix + MTP soak (40% cache hit, 86.9% KV, zero OOB or asserts). This lifts the prefix-caching-vs-spec-decode either/or on recent nightlies, with the caveat that vLLM marks the align path experimental. See the field notes’ 2026-08-05 update for the run detail.
2026-08-05: #31424 (NGC container lag) closed not_planned: the vendor-container lag is now permanent-by-neglect.
Re-verified 2026-08-05: §4.4’s b12x auto-selection exclusion still holds, checked in _POSSIBLE_NVFP4_KERNELS (dense) and the MoE oracle at v0.26.0 and main; b12x is excluded from auto on both paths and the flashinfer-ai/flashinfer#3134 guard has not landed. Still not re-audited, re-check before relying on: §7.2–7.3 (backend auto-selection order) and §10 (image and dependency pins).

The scorecard: what worked, what didn’t, what we found

We adopted the blog’s recipe in one commit (2026-06-02, one day after publication) and have run it under 24/7 agent traffic since. The verdict, with receipts:

What worked

The cu130-nightly image track: adopted as a gitsha-pinned nightly (0.22.1rc1.dev26, CUDA 13.0.2), per the blog’s own “pin a digest” clause. It unlocked native FP4 immediately: --moe-backend auto → FLASHINFER_CUTLASS, ~55.8–58.5 tok/s vs ~50 on forced Marlin (2026-06-02, batch-1 decode). Still the fleet’s serving basis today.
Trusting auto backend selection, on that image track. The blog’s claim that “recent FlashInfer CUTLASS paths are much stronger than older Spark guidance suggests” measured out on our boxes the same day.
CUDA graphs enabled. We retired --enforce-eager fleet-wide the same commit (it had been a v0.20.0 Mamba graph-capture workaround, §8) and never looked back.
Built-in --reasoning-parser nemotron_v3 over NVIDIA’s HF-card parser plugin: simpler, coherent, no plugin file.
The JIT pre-warm ping (max_tokens: 3 at boot): we run the same trick the blog describes.

What didn’t, or didn’t survive production

auto on the images that predated the blog. On cu129/v0.20.0 images, auto crashed; our cards from that era read “cu129 stock: auto crashes; marlin correct.” The blog’s advice was true of the image track it tested and wrong for the installed base that preceded it. Reconciling that (blog vs. repo vs. our logs) is how we learned it was a version gap, not a dispute, and it’s why every claim in this report carries a commit and date.
The demo-shaped numeric defaults. --gpu-memory-utilization 0.85 matches our own single-model figure, but co-residency forced 0.3–0.4, and taught us the unified-memory rule the hard way: the GPU reservation is unswappable, so the reservation itself is the memory pressure (0.6 ≈ 71 GiB got our 35B OOM-killed on 2026-07-03). --max-num-seqs 4 and --max-model-len 131072 became 4/8/16 and 32K/262K per box role. None of these are errors in the blog; they’re demo-shaped numbers that don’t transfer to a multi-model production box.
MTP as “evaluate.” We productionized --speculative-config MTP and paid the tuition: 86.8% acceptance in validation, then CUDA illegal memory access every few hours only under concurrent load (upstream #41190/#42084). The configuration that avoided it, a genuine W4A4 checkpoint putting experts and drafter on the CUTLASS path, is the report’s biggest self-discovered lesson (below). The blog’s “validate before relying on it” clause was fully vindicated; batch-1 validation was not enough.
The fp8 KV-cache caution: inverted by measurement. The blog frames --kv-cache-dtype fp8 as a memory-pressure concession (“may affect predictability… noticeable performance cost… avoid unless memory pressure requires it and quality checks pass”). We ran the quality check it prescribes (2026-07-20, A/B on the production 35B, temp-0 task-scored extraction + greedy probes): byte-identical extraction output across arms 10/10 docs, fp8 more deterministic intra-arm (10/10 vs 9/10), and ~27% faster decode (96.6 vs 76.3 tok/s, halved KV traffic on a bandwidth-bound box, with MTP re-reading KV every verification step), plus double the 262K-context capacity in the same pool (~3.2 vs ~1.6 sequences). Not an error in the blog (its escape clause is exactly what we executed), but on decode-bound long-context serving the cost-benefit framing runs the other way: fp8 KV is a default with a per-checkpoint validation step, not a concession.

What we found that the blog never mentions

The checkpoint’s quant format decides the FP4 backend, not your flags (W4A4 → native CUTLASS; W4A16 → Marlin fallback). The load-bearing rule of this entire report (§6), and the actual fix for the MTP crashes.
b12x is opt-in, not auto at the audited commit (§4.4): auto cannot select the Spark-native MoE path the blog’s era shipped.
Backend flags the recipe needs but the blog never names: --attention-backend (TRITON_ATTN / flashinfer per model), --mamba-ssm-cache-dtype float32 for hybrid-Mamba, --max-num-batched-tokens prefill chunking, and --async-scheduling, including the discovery that spec-decode configurations can silently disable async scheduling and tax prefill (§the n-gram lesson in our field notes).
Coordinator interactions: behind a llamactl-style proxy, --served-model-name 404s every request; the blog’s standalone recipe breaks unmodified behind a gateway.
Unified-memory operational arithmetic: KV-pool math from the boot log (per-token cost, boot floor ≈ 0.24 at 262K ctx, ≥15 GiB host headroom) that turns --gpu-memory-utilization from folklore into subtraction.

The DGX Spark architecture
Why DGX Spark has been hard for vLLM
CUTLASS: the templated GEMM engine
FlashInfer: the kernel-serving layer
Marlin: the universal fallback
NVFP4 / MXFP4 / FP8 quantization on Spark
How vLLM does MoE, and how it lands on Spark
CUDA graphs and --enforce-eager
The history: how Spark support was built
Prebuilt containers vs. rolling your own
Appendix: glossary and file index

1. The DGX Spark architecture

1.1 What the box is

The DGX Spark is built on the NVIDIA GB10 Grace Blackwell superchip, a Grace ARM64/aarch64 CPU and a Blackwell GPU fused onto one package with a 128 GB unified LPDDR5X memory pool shared between CPU and GPU. This single fact drives most of the operational quirks: there is no separate “VRAM”; weights, KV cache, the OS page cache, and the container runtime all draw from the same 128 GB. Memory bandwidth (~273 GB/s class) is far below a datacenter card, which is why decode throughput is the binding constraint on Spark, not compute.

vLLM recognizes the unified-memory nature of the platform explicitly:

vllm/utils/mem_utils.py:110: “On UMA (Unified Memory Architecture) platforms where CPU and GPU share physical memory (e.g. GH200, DGX Spark, Jetson Orin)…”
vllm/platforms/interface.py:799: is_integrated_gpu() documents DGX Spark as a UMA device.

1.2 The compute capability that matters: `sm_121`

The GB10 GPU reports compute capability 12.1, i.e. sm_121 (architecture-specific target sm_121a). This is the single most important number in this entire document. It places Spark in the “SM12x family” of consumer/workstation Blackwell parts:

Part	Compute capability	`nvcc` target	Class
RTX 5090	12.0	`sm_120a`	consumer Blackwell
RTX PRO 6000	12.0	`sm_120a`	workstation Blackwell
DGX Spark (GB10)	12.1	`sm_121a`	workstation Blackwell
B200 / GB200	10.0	`sm_100a`	datacenter Blackwell

vLLM’s own source documents the mapping in two kernel generators:

csrc/quantization/marlin/generate_kernels.py:18 and csrc/moe/marlin_moe_wna16/generate_kernels.py:18: # SM89 and the SM12x family (SM120 RTX 5090, SM121 DGX Spark GB10)

1.3 The trap: `sm_121` is not `sm_100`

Datacenter Blackwell (sm_100) and workstation Blackwell (sm_120/sm_121) are different architectures with different tensor-core instruction sets, even though both are marketed as “Blackwell.” A kernel compiled and tuned for sm_100 will not run on sm_121. In CUTLASS terms they require entirely separate kernel instantiations: different BlockwiseScaleConfig, different warp-specialized kernel schedules, and different cutlass::arch::Sm1xx tags (see §3). The practical consequence: almost all generic “Blackwell NVFP4” guidance on the internet is written for sm_100 and does not transfer to Spark. This is the root cause of nearly every Spark gotcha.

1.4 What the tensor cores natively support

The Marlin generator records the key hardware fact about the SM12x family:

csrc/quantization/marlin/generate_kernels.py:18-23:

# SM89 and the SM12x family (SM120 RTX 5090, SM121 DGX Spark GB10)
# fully support mma.sync.aligned.m16n8k32.row.col.f32.e4m3.e4m3.f32.
# SM90 and SM100 can use this PTX, but it's simulated
# with FP16 MMA, so it cannot achieve any acceleration.
if arch == 89 or arch // 10 == 12:
    SUPPORT_FP8 = True

So sm_121 has native FP8 (e4m3) tensor-core MMA, and (via OpClassBlockScaledTensorOp) native block-scaled FP4 tensor cores. The hardware is capable; the historical problems were entirely about software coverage for this specific architecture.

2. Why DGX Spark has been hard for vLLM

Four compounding reasons:

1. Architecture fragmentation + arch-specific cubins. Modern quantized GEMM kernels (CUTLASS, FlashInfer’s CuteDSL kernels) are compiled into architecture-specific cubins. A wheel that ships sm_100a and sm_120a cubins but not sm_121a will fail at runtime on Spark, not with a graceful fallback, but with a hard error or, worse, silent garbage. The canonical symptom:

csrc/.../fp4/nvfp4_scaled_mm_entry.cu:66-68 (runtime SM dispatch):

STD_TORCH_CHECK_NOT_IMPLEMENTED(
    false, "No compiled nvfp4 mm kernel for SM ", sm,
    ". Recompile with CUDA >= 12.8 and CC >= 100.");

2. sm_121 is the newest, least-covered target. sm_120 (RTX 5090) support landed in vLLM in mid-2025; sm_121 is a strict latecomer that had to be threaded through every kernel’s arch list one at a time (see §9). For a long window, sm_121 was partially covered: some kernels had it, others silently excluded it, producing inconsistent behavior.

3. aarch64. Spark is ARM64. Historically the CUDA Python ecosystem was x86-first; aarch64 CUDA wheels were patchy. This is now largely solved (PyTorch ≥2.11.0 ships aarch64 CUDA wheels on PyPI, and vLLM publishes aarch64 wheels), but it added friction during the early Spark period and still bites anyone on older torch.

4. CUDA toolkit floor. sm_121 kernels require CUDA ≥ 12.8 to even compile; CUDA 13.0 is preferred (it introduces family targets, see §3.4). Building against an older toolkit silently drops Spark from the arch list:

CMakeLists.txt:99-110:

if(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 13.0)
  set(CUDA_SUPPORTED_ARCHS "7.5;8.0;8.6;8.7;8.9;9.0;10.0;11.0;12.0")   # family targets
elseif(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.8)
  set(CUDA_SUPPORTED_ARCHS "7.5;8.0;8.6;8.7;8.9;9.0;10.0;10.1;10.3;12.0;12.1")
else()
  set(CUDA_SUPPORTED_ARCHS "7.0;7.5;8.0;8.6;8.7;8.9;9.0")               # no sm_121 at all
endif()

The rest of this guide explains the moving parts that these four problems touch.

3. CUTLASS: the templated GEMM engine

3.1 What CUTLASS is and why it matters

CUTLASS (CUDA Templates for Linear Algebra Subroutines) is NVIDIA’s open-source C++ template library for building high-performance GEMMs directly on the tensor cores. vLLM uses CUTLASS 3.x (“c3x”) to implement its first-choice quantized matmuls: FP8 W8A8, NVFP4 W4A4, and the blockwise-scaled variants. When CUTLASS works for your arch + dtype, it is typically the fastest path, which is why the backend oracles try it before falling back to Marlin.

Key files:

csrc/quantization/w8a8/cutlass/scaled_mm_c3x_sm120.cu: FP8/INT8 scaled-MM entry for SM120/121
csrc/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8_dispatch.cuh: per-tensor FP8, with M-shape-based tile selection
csrc/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm120_fp8_dispatch.cuh: blockwise FP8
csrc/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu: NVFP4 GEMM
csrc/quantization/fp4/nvfp4_blockwise_moe_kernel.cu: NVFP4 blockwise MoE
csrc/quantization/fp4/nvfp4_scaled_mm_entry.cu: runtime SM dispatch (the source of the “No compiled nvfp4 mm kernel for SM …” error)

3.2 Why CUTLASS needs separate kernels for `sm_100` vs `sm_120`/`sm_121`

CUTLASS kernels are parameterized by an architecture tag that selects the MMA atoms and the warp-specialized scheduling strategy. Datacenter and workstation Blackwell get different tags and different schedules:

Aspect	`sm_100` (datacenter)	`sm_120`/`sm_121` (workstation)
Arch tag	`cutlass::arch::Sm100`	`cutlass::arch::Sm120`
Blockwise scale config	`Sm100BlockwiseScaleConfig`	`Sm120BlockwiseScaleConfig`
Kernel schedule	`KernelTmaWarpSpecializedBlockwise{1,2}SmSm100`	`KernelTmaWarpSpecializedBlockwise{Pingpong,Cooperative}Sm120`

These are distinct compiled kernels; there is no shared binary. That is exactly why “it works on my B200” tells you nothing about Spark.

3.3 The `enable_sm120_family` guard: one binary for SM120 and SM121

The mechanism that lets a single compiled kernel serve both RTX-class (sm_120) and Spark (sm_121) is a thin device-side wrapper:

csrc/cutlass_extensions/common.hpp (≈ lines 128-142):

// SM12x family includes SM120 (RTX 5090) and SM121 (DGX Spark GB10)
template <typename Kernel>
struct enable_sm120_family : Kernel {
  template <typename... Args>
  CUTLASS_DEVICE void operator()(Args&&... args) {
#if defined __CUDA_ARCH__
  #if (__CUDA_ARCH__ >= 1200 && __CUDA_ARCH__ < 1300)
    Kernel::operator()(std::forward<Args>(args)...);
  #else
    printf("This kernel only supports sm120f.\n");
    asm("trap;");
  #endif
#endif
  }
};

The range check __CUDA_ARCH__ >= 1200 && < 1300 admits 1200 (sm_120) and 1210 (sm_121), the whole SM12x family. Several Spark bugs over the past months were precisely cases where a kernel used a narrower sm_120-only guard and thus trapped or fell through on sm_121; the fixes consisted of routing those kernels through enable_sm120_family (e.g. the per-tensor FP8 dispatch fix, PR #41215).

3.4 The `a` vs `f` suffix, and `cuda_archs_loose_intersection`

nvcc arch targets carry a suffix:

12.1a (architecture-specific): one cubin for exactly sm_121. To cover the family you must list every member: 12.0a;12.1a.
12.0f (family): one cubin that runs across the whole SM12x family: sm_120, sm_121, and future SM12x. Requires CUDA ≥ 13.0. (See NVIDIA’s family-specific-architecture docs.)

vLLM’s CMake picks the right form by toolkit version. For example, scaled-MM:

CMakeLists.txt:755-758:

if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
  cuda_archs_loose_intersection(SCALED_MM_ARCHS "12.0f" "${CUDA_ARCHS}")
else()
  cuda_archs_loose_intersection(SCALED_MM_ARCHS "12.0a;12.1a" "${CUDA_ARCHS}")
endif()

The same pattern appears for FP4_ARCHS, MLA_ARCHS, MARLIN_*_ARCHS, etc. The helper cuda_archs_loose_intersection (cmake/utils.cmake:368-416) is what makes the family form match a user’s specific arch: given source 12.0f and a target list containing 12.1a, it strips suffixes, compares the major version (both 12), and matches, so a single 12.0f build covers Spark. This family-level matching (added in PR #38126, “Fix DGX Spark logic”) is why a CUDA-13 build with just 12.0f correctly serves sm_121.

Operator takeaway: if you build from source on CUDA 12.8/12.9 you must list 12.1a (or 12.0a;12.1a); on CUDA 13.0+ 12.0f suffices. The in-repo Docker Grace-Blackwell example uses '9.0 10.0+PTX'. That is for GH200/GB200 and is wrong for GB10; you must override it.

4. FlashInfer: the kernel-serving layer

4.1 What FlashInfer is in vLLM

FlashInfer is a kernel library that vLLM leans on heavily for inference-optimized kernels. vLLM’s wrapper is vllm/utils/flashinfer.py (~1000 lines). Surface area spans:

Attention: paged/ragged batch decode & prefill, TRT-LLM attention backends, MLA (DeepSeek), FP4 KV cache.
Fused MoE: TRT-LLM BF16/FP8/FP4 block-scale MoE, CUTLASS fused MoE, CuteDSL NVFP4 MoE, and the SM12x b12x fused MoE.
Quantized GEMM: FP4/MXFP4 quantize, NVFP4 scaled MM, block-scale FP8.
RoPE and collective all-reduce (MNNVL/NVLink).

4.2 How FlashInfer ships kernels: prebuilt cubins vs. JIT

This is the crux of FlashInfer’s relationship with Spark. FlashInfer kernels arrive one of two ways (vllm/utils/flashinfer.py:27-63):

Prebuilt cubins: the flashinfer-cubin package (pinned flashinfer-cubin==0.6.11.post2 in requirements/cuda.txt:13) downloads precompiled cubins from an NVIDIA artifact repository. Fast, but only covers the arches NVIDIA chose to build.

JIT: if cubins are missing, FlashInfer compiles kernels at first request, which requires nvcc to be present in the container:

if not has_flashinfer_cubin() and shutil.which("nvcc") is None:
    # FlashInfer unavailable: no cubins and no nvcc to JIT
    return False

Why Spark broke: for a long while the prebuilt FlashInfer wheels shipped zero sm_120/sm_121 cubins, and stock inference images don’t carry nvcc (it’s in the devel image, not runtime). So FlashInfer’s FP4 path either errored (Failed to run cutlass FP4 gemm on sm120/sm121 … Error Internal) or (the dangerous case) produced incoherent tokens with no error. This is the original sin behind the field rule “force Marlin on Spark.”

4.3 `b12x`: the modern SM12x MoE / FP4 path

The newer, Spark-native path is b12x (FlashInfer’s SM12x block-scaled GEMM, built on CuteDSL, the CUTLASS C++ tensor-layout DSL). It exists in two forms:

MoE expert: vllm/model_executor/layers/fused_moe/experts/flashinfer_b12x_moe.py:

“FlashInfer CuteDSL fused MoE expert for SM12x (SM120/SM121, RTX Pro 6000 / DGX Spark). Uses b12x_fused_moe … which fuses token dispatch, two GEMMs, SwiGLU activation, and topk-weight reduction into a single kernel call. Input quantization (BF16→FP4) is performed inside the kernel … Only NVFP4 … is supported.”

Notably, it takes BF16 hidden states directly and quantizes inside the kernel; scale factors are pre-converted to MMA layout once in process_weights_after_loading.
Dense linear: vllm/model_executor/kernels/linear/nvfp4/flashinfer.py:221:

class FlashInferB12xNvFp4LinearKernel(NvFp4LinearKernel): """NVFP4 GEMM via FlashInfer's b12x CuTe DSL warp-level MMA kernel (SM120+)."""

Both require CC ≥ 12.0. b12x landed in vLLM in PR #40082 (first released in v0.22.0).

4.4 The critical nuance: `b12x` is opt-in, not auto `[verified 2026-06-02: highest-churn fact; re-check each release]`

This is the single most under-appreciated fact for Spark operators, and it refines the common “just leave backends on auto” advice. The official blog recommends leaving backend selection on auto, but at the audited vLLM main commit (689e…, 2026-06-02), b12x is deliberately excluded from auto-selection until an upstream CUTLASS sm_121 MMA-guard issue (flashinfer-ai/flashinfer#3134) is resolved:

vllm/model_executor/layers/fused_moe/oracle/nvfp4.py:170-173:

# NOTE: the kernels are selected in the following order.
# FLASHINFER_B12X is intentionally excluded from auto-selection until
# the upstream CUTLASS SM121 MMA op guard is resolved; use
# moe_backend="flashinfer_b12x" to opt in explicitly.

and the matching note for dense linears in vllm/model_executor/kernels/linear/__init__.py:383-395:

# FlashInferB12xNvFp4LinearKernel excluded from auto-selection until
# upstream CUTLASS SM121 MMA op guard is resolved; use
# VLLM_NVFP4_GEMM_BACKEND=flashinfer-b12x to opt in explicitly.

So on that code path, --moe-backend auto will not pick the Spark-native b12x kernel; it walks the list below (§7) and lands on whatever else supports sm_121. To actually use b12x you pass --moe-backend flashinfer_b12x (and/or --linear-backend flashinfer-b12x). This is why the audited-commit Spark recipe is either “force Marlin,” “explicitly opt into b12x and validate,” or, as our production fleet later confirmed, get onto the FLASHINFER_CUTLASS native path with a genuine W4A4 checkpoint. Re-check this against the exact release you deploy: it is expected to change once the upstream guard lands.

Update [verified 2026-08-10]: the exclusion still holds at v0.26.0 and main, but two details in the quoted source comments have since moved — treat the comments above as a 2026-06-02 snapshot:

The env-var opt-in is gone. VLLM_NVFP4_GEMM_BACKEND was deprecated in v0.23 (deprecated_env(..., "v0.23", "Use --linear-backend.")) and removed in v0.25. On any current image, VLLM_NVFP4_GEMM_BACKEND=flashinfer-b12x is a silent no-op — stale recipes still hand it out. The only surviving opt-in is the CLI value: --linear-backend flashinfer_b12x / --moe-backend flashinfer_b12x.

The exclusion is global, not sm_121-scoped. FLASHINFER_B12X is absent from AVAILABLE_BACKENDS on every GPU — an sm_120 RTX PRO 6000 also falls to Marlin under auto. sm_121 is the reason cited in the comment, not the scope. The still-open PR #47577 would re-add auto-selection for sm_120 only and deliberately keep sm_121 on the explicit opt-in — auto keeps skipping b12x and falling through to whatever the image + checkpoint actually support (FLASHINFER_CUTLASS on a genuine W4A4 checkpoint with sm_121 cubins, Marlin on a W4A16 checkpoint — see §4.4), not an unconditional Marlin default. So b12x remains opt-in-only, never auto-selected on a Spark, in every released version — and forcing it currently carries a cluster of open bugs (activation memory roughly doubling → ~55% less KV cache, #49079; an Xid 31 MMU fault under concurrent chunked prefill, #50189).

4.5 `flashinfer_cutlass_moe` vs `flashinfer_b12x_moe`

	`flashinfer_cutlass_moe` (older)	`flashinfer_b12x_moe` (newer)
Dispatch	per-block GEMM	single fused kernel (dispatch + 2 GEMMs + SwiGLU + reduce)
Quant	FP8 block, MXFP4, static NVFP4	NVFP4 only
Input	pre-quantized activations	BF16 (quantize inside kernel)
SM target	SM90 / SM100(+)	SM120+
Auto-selected?	yes (in the list)	no, opt-in

PR #39825 (“Disable FlashInfer CUTLASS MoE on SM121”) was the stopgap that kept the older path from misbehaving on Spark before b12x existed.

5. Marlin: the universal fallback

5.1 What Marlin is

Marlin (originating from IST-DASLab, integrated via Neural Magic; csrc/quantization/marlin/marlin.cu) is a mixed-precision, weight-only quantized GEMM: 16-bit activations (FP16/BF16) multiplied by low-bit weights. The “wna16” naming = weight n-bit, activation 16-bit. It supports a broad matrix of weight formats (csrc/quantization/marlin/generate_kernels.py:29-100): AWQ-INT4, GPTQ-INT4/INT8, FP8 (kFE4M3fn), NVFP4 (kFE2M1f weights, kFE4M3fn scales), MXFP4, and MXFP8.

5.2 Why Marlin is the robust fallback on Spark

Two structural reasons:

1. It is template-generated, not arch-pinned. Marlin emits kernels from Jinja2 templates parameterized by thread/tile/group config, so the same source covers sm_80, sm_89, sm_120, sm_121 without bespoke per-arch kernel files the way CUTLASS does. Its CMake arch lists explicitly include the SM12x family:

CMakeLists.txt: MARLIN_FP8_ARCHS = "8.9;12.0a;12.1a" (CUDA<13) / "8.9;12.0f" (CUDA≥13); MARLIN_MOE_FP8_ARCHS = "8.9;12.0;12.1".

2. It covers both dense linears and MoE. Dense: csrc/quantization/marlin/. MoE: csrc/moe/marlin_moe_wna16/. So forcing Marlin fixes the whole model in one move, which is exactly why it became the Spark workaround when CUTLASS/FlashInfer FP4 paths were broken for both the dense and MoE halves of a model.

5.3 Marlin for NVFP4

Marlin has a real FP4 path. The NVFP4 linear kernel is MarlinNvFp4LinearKernel (“NVFP4 weight-only GEMM via Marlin (W4A16)”, vllm/model_executor/kernels/linear/nvfp4/marlin.py:18). It performs a non-trivial scale repacking, converting the FP8-E4M3 block scales into Marlin’s internal “S0E5M3” dequant layout (vllm/.../utils/marlin_utils_fp4.py:53-114). You route to it explicitly with VLLM_NVFP4_GEMM_BACKEND=marlin (env, deprecated per the blog) or, preferably, the CLI --linear-backend marlin / --moe-backend marlin.

5.4 Where Marlin sits in selection, and #40923

In the NVFP4 MoE oracle (§7) Marlin sits near the end of the auto list, a deliberate “always works, decent speed” backstop after the CUTLASS/FlashInfer attempts. PR #40923 (“Marlin MoE: include SM 12.x in default arch list”) added the SM12x family to the default Marlin-MoE build arch list so stock builds carry Spark Marlin-MoE kernels out of the box (currently only in v0.22.1rc0).

Cost of Marlin: as a weight-only W*A16 kernel it dequantizes weights to 16-bit and runs the matmul in 16-bit, so it does not exploit Spark’s native FP4/FP8 tensor cores as fully as a true block-scaled CUTLASS/b12x kernel can. On a bandwidth-bound box like Spark this is often acceptable, but it’s the reason Marlin is the fallback, not the goal.

6. NVFP4 / MXFP4 / FP8 quantization on Spark

6.1 NVFP4 format

NVFP4 is a 4-bit float with two-level block scaling:

Elements: e2m1 (2-bit exponent, 1-bit mantissa; max 6.0), packed 2-per-byte.
Per-block scale: FP8 e4m3, one per 16 contiguous elements along K.
Global scale: an fp32 scalar (one for weights, one for activations); the kernel combines them as alpha = 1 / (input_global_scale * weight_global_scale).

So the effective value is element × fp8_block_scale × fp32_global_scale. (See vllm/.../quantization/modelopt.py around the NVFP4 method, and csrc/.../fp4/nvfp4_utils.cuh.) This block structure is why NVFP4 needs block-scaled tensor-core MMA (OpClassBlockScaledTensorOp) to run fast, which only sm_100/sm_120/sm_121 provide, in arch-specific form.

6.2 The dense-vs-MoE split: the gotcha that wasted everyone’s time

NVFP4 weights appear in two independent code paths:

Dense linear layers → ModelOptNvFp4LinearMethod, served by the linear kernel oracle (cutlass_scaled_fp4_mm, FlashInfer, or Marlin).
MoE expert weights → the fused-MoE oracle (§7).

These are selected independently. The historical Spark trap: the model-card env vars VLLM_USE_FLASHINFER_MOE_FP4=1 / VLLM_FLASHINFER_MOE_BACKEND=… only reroute the MoE half. The dense NVFP4 linears keep going through the broken CUTLASS FP4 path and crash or emit garbage. The only single-switch fix that covers both halves is Marlin (--linear-backend marlin --moe-backend marlin), or, on current images, explicitly opting both into b12x.

6.3 The kernels that consume NVFP4, and the runtime SM dispatch

The CUTLASS FP4 entry point dispatches on the runtime SM and fails loudly if the matching cubin wasn’t compiled in:

csrc/.../fp4/nvfp4_scaled_mm_entry.cu (≈ 48-68):

const int32_t sm = get_sm_version_num();
#if defined(ENABLE_NVFP4_SM100) ...
  if (sm >= 100 && sm < 120) { cutlass_scaled_fp4_mm_sm100a(...); return; }
#endif
#if defined(ENABLE_NVFP4_SM120) ...
  if (sm >= 120 && sm < 130) { cutlass_scaled_fp4_mm_sm120a(...); return; }   // SM120/121
#endif
  STD_TORCH_CHECK_NOT_IMPLEMENTED(false, "No compiled nvfp4 mm kernel for SM ", sm, ...);

This device guard + runtime dispatch was added/hardened in PR #29711. The lesson: a wheel built without ENABLE_NVFP4_SM120 / the 12.1a arch simply has no Spark FP4 kernel, and you hit the NOT_IMPLEMENTED at first FP4 matmul.

6.4 NVFP4 vs MXFP4 (and who uses which)

	NVFP4	MXFP4
Element	`e2m1` (4-bit)	`e2m1` (4-bit)
Block size (K)	16	32
Block scale	`FP8 e4m3`	`e8m0` (exponent-only)
Typical models	Qwen, Nemotron-3	GPT-OSS (MoE)

MXFP4 (vllm/.../quantization/mxfp4.py) is what the GPT-OSS family ships in. The block-scaled FP4/MXFP4 CUTLASS kernels for sm_120/sm_121 are partly provided through QuTLASS (cmake/external_projects/qutlass.cmake, fetched from IST-DASLab; requires CUDA 12.8+ and CC 10.0f/12.0f).

6.5 FP8: the path of least resistance on Spark

Because sm_121 has native e4m3 FP8 MMA (§1.4) and FP8 is covered by both CUTLASS (scaled_mm_sm120_fp8_dispatch) and Marlin, FP8 is usually far less fragile than NVFP4 on Spark; on the audited paths it needed no Marlin dance. Treat that as image- and model-dependent rather than a law (Spark FP8 failures have surfaced in specific nightlies and models); smoke-test the exact model and container before relying on auto. On a 128 GB unified box the memory saving of FP4 over FP8 is often irrelevant for the models people actually run, so a practical rule is: prefer FP8 (or BF16) unless a model is only published in NVFP4/MXFP4.

7. How vLLM does MoE, and how it lands on Spark

7.1 The modular fused-MoE architecture

vLLM decomposes a fused-MoE layer into composable stages (vllm/model_executor/layers/fused_moe/modular_kernel.py:45-79):

[Router] → [Quantize-Dispatch] → [Permute-Experts-Unpermute] → [Combine]

implemented via three classes: FusedMoEPrepareAndFinalizeModular (prep + finalize), FusedMoEExpertsModular (the expert GEMMs + activation + optional quant), and FusedMoEModularKernel (the glue). Each concrete backend is an “experts” implementation under vllm/model_executor/layers/fused_moe/experts/ (e.g. flashinfer_b12x_moe.py, marlin_moe.py, triton_moe.py, flashinfer_cutlass_moe.py).

7.2 The oracle: how a backend is chosen `[verified 2026-06-02]`

For NVFP4 MoE, select_nvfp4_moe_backend() (oracle/nvfp4.py:160-333) walks a priority list and asks each backend whether it supports the current config/device:

AVAILABLE_BACKENDS = [
    NvFp4MoeBackend.FLASHINFER_TRTLLM,
    NvFp4MoeBackend.FLASHINFER_CUTEDSL,
    NvFp4MoeBackend.FLASHINFER_CUTEDSL_BATCHED,
    NvFp4MoeBackend.FLASHINFER_CUTLASS,
    NvFp4MoeBackend.VLLM_CUTLASS,
    NvFp4MoeBackend.MARLIN,
    NvFp4MoeBackend.EMULATION,
]
# FLASHINFER_B12X intentionally excluded from auto-selection (opt in explicitly)

The --moe-backend CLI value maps onto this enum in map_nvfp4_backend() (oracle/nvfp4.py:141-157), and the available options + their docstrings (including “flashinfer_b12x: … for SM12x (RTX Pro 6000 / DGX Spark)”) live in vllm/config/kernel.py:171-190.

7.3 What this means on Spark

--moe-backend auto on sm_121 will skip b12x (opt-in only, at the audited commit, §4.4) and try TRT-LLM / CuteDSL / CUTLASS paths, then VLLM_CUTLASS, then Marlin, then emulation. Whether the earlier entries actually run on sm_121 depends on the image’s cubins; this is why auto behavior on Spark has historically been a coin-flip and why people pinned Marlin.
For a predictable Spark deployment you have two solid choices:
1. Force Marlin for both linear and MoE: guaranteed-correct, slightly slower.
2. Explicitly opt into b12x (--moe-backend flashinfer_b12x, --linear-backend flashinfer-b12x) on a v0.22.0+ image (the fast native path) and validate output coherence before trusting it (the b12x exclusion-from-auto exists precisely because the upstream guard isn’t fully settled).

8. CUDA graphs and `--enforce-eager`

8.1 What CUDA graphs do, and what `enforce_eager` turns off

vLLM captures the decode forward pass into a CUDA graph so the GPU replays a recorded sequence of kernel launches with near-zero per-launch CPU overhead. enforce_eager disables this entirely:

vllm/config/model.py: enforce_eager: bool = False: “If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility.”

When set, vLLM forces cudagraph_mode = NONE and clears the capture-size list.

8.2 Piecewise vs. full capture (and the V1 default)

vllm/config/compilation.py:599-634 defines the modes:

NONE: no capture.
PIECEWISE: capture everything except graph-incompatible ops (e.g. some attention), which run outside the graph for flexibility.
FULL / FULL_DECODE_ONLY.
FULL_AND_PIECEWISE: full graphs for decode batches, piecewise for prefill/mixed. This is the V1 default and the most performant for most models.

Graphs are captured at a discrete set of batch sizes (cudagraph_capture_sizes), auto-generated roughly as [1,2,4] + range(8,256,8) + range(256, max, 16), with max_cudagraph_capture_size defaulting to min(max_num_seqs*2, 512). This is the mechanism behind the field trick of capping --max-num-seqs to bound the capture set.

8.3 Why Mamba / hybrid (`nemotron_h`) models need eager on Spark

First, the scope: the official blog’s benchmark path kept CUDA graphs enabled; this section covers a separate field failure mode, not a blanket Spark rule. Hybrid SSM/Mamba models maintain recurrent state and use Triton kernels with persistent in-place state updates. Graph capture freezes kernel-launch semantics and pointer bindings, which collides with that dynamic state; hence Mamba models implement special copy_inputs_before_cuda_graphs / get_seqlen_agnostic_capture_inputs hooks (vllm/model_executor/models/mamba.py:270-274, mamba2.py, jamba.py, nemotron_h.py). On sm_121 specifically, graph capture of the Mamba-2 Triton path has surfaced an “illegal instruction” during capture in our deployments, so the operational fix has been --enforce-eager, applied only when a specific model/image combination shows graph-capture instability, and documented with the tested model, image, CUDA version, and error.

8.4 Why eager is a crutch, not a destination

enforce_eager pays for itself in launch overhead on every decode step; field measurements on the Nemotron hybrid models put the cost around ~37% throughput on Spark, and the broader range is roughly 5–30% depending on batch size and model depth. It’s an escape hatch for a kernel/capture incompatibility, not a tuning choice. The right long-term fix is upstream: make the Mamba-2 Triton kernels capture-safe on sm_121 (or provide a CUDA-graph-compatible SSM path), at which point Spark hybrid models can drop --enforce-eager and recover that throughput. Until then it remains model-specific: apply it to nemotron_h/Mamba hybrids, not to dense or standard-MoE models that capture fine.

9. The history: how Spark support was built

Support arrived in two waves: RTX-class sm_120 first, then Spark sm_121 threaded in behind it. Release attributions below are verified with git tag --contains [verified 2026-06-02].

Wave 1, sm_120 (RTX Blackwell) groundwork, mid-2025:

#17280 CUTLASS W8A8 FP8 for sm_120; #21309 NVFP4 W4A4 for sm_120; #22131 block FP8; #29242 NVFP4 MoE CUTLASS; #29711 device guard + runtime SM dispatch for cutlass_scaled_fp4_mm.

Wave 2, sm_121 (DGX Spark) era, 2026:

PR	What	First release
#33517 (2026-02-06)	`enable_sm120_family` template: first explicit DGX Spark commit; one binary for SM120+SM121	early 2026
#38126 (2026-03-27)	“Fix DGX Spark logic”: CMake family matching (`12.0f`↔`12.1a`), arch lists gain `12.1`	v0.19.0
#38423 (2026-03-30)	NVFP4 bugfixes for DGX Spark + RTX50	(post-v0.19)
#39825 (2026-04-14)	Disable FlashInfer CUTLASS MoE on SM121 (stopgap)	v0.20.0
#35568 (2026-05-15)	Fix incorrect SM121 exclusion from Marlin/CUTLASS FP8 paths	v0.22.0
#40082 (2026-05-20)	Integrate FlashInfer `b12x` MoE + FP4 GEMM for SM120/121	v0.22.0
#41215 (2026-05-20)	Route per-tensor FP8 through `enable_sm120_family` on SM12.1	v0.22.0
#40923 (2026-05-28)	Marlin MoE: include SM12.x in default arch list	v0.22.1rc0

The key takeaway: the cluster of fixes that make Spark coherent end-to-end, the SM121 FP8 exclusion fix (#35568) and the b12x integration (#40082), first ship together in v0.22.0 (2026-05-29). Anything older (including the popular v0.20.0) lacks b12x and the exclusion fix; it can be made to work only by forcing Marlin. Release cadence is ~2–3 weeks.

10. Prebuilt containers vs. rolling your own

10.1 Decision summary `[verified 2026-06-02: image tags & version floors drift; re-check each release]`

You are…	Do this
Running stock models via container (most people)	Use a pinned digest of an image you have smoke-tested on Spark. `v0.22.0` is the first stable baseline in this audit containing `b12x` (#40082) + the SM121 FP8 exclusion fix (#35568), a historical minimum, not the July 2026 recommendation; newer stable or nightly images may be preferable but must be re-audited for backend selection and cubin coverage (see the update timeline: the cu129 v0.25.1 stable still shipped no sm_121 cubins on 2026-07-19).
Tracking the newest kernels / want #40923 Marlin-MoE arch	`vllm/vllm-openai:v0.22.1rc0` or the blog’s compatibility track `cu130-nightly` (pin a digest; it’s CUDA-13).
Modifying C++/CUDA, or need a kernel no image ships	Build from source with `TORCH_CUDA_ARCH_LIST=12.1a`, CUDA ≥ 12.8 (13.0 preferred).
On an old image (≤ v0.21) you can’t change	Force Marlin (`--moe-backend marlin --linear-backend marlin`); expect dense+MoE to need it.

You almost never need a from-scratch native compile on the Spark itself unless you’re editing kernels; the published aarch64 images/wheels exist precisely so you don’t. From-source is the path for kernel work or when a needed fix is on main but not yet released.

10.2 If you build your own: the arch traps

List the right arch. CUDA 12.8/12.9 → 12.1a (or 12.0a;12.1a). CUDA 13.0+ → 12.0f (family target covers Spark). Below 12.8, sm_121 cannot be built at all (CMakeLists.txt:99-110).
Do not copy the in-repo Grace-Blackwell Docker example’s torch_cuda_arch_list='9.0 10.0+PTX'. That targets GH200/GB200, not GB10. Override it with 12.1a/12.0f.
Match the dependency pins to the revision you build. Current main (requirements/cuda.txt) pins flashinfer-python==0.6.11.post2, flashinfer-cubin==0.6.11.post2, and nvidia-cutlass-dsl[cu13]==4.5.2 [verified 2026-06-02: these pins move; read the pin in the exact tag you build]. (Older field notes citing cutlass-dsl 4.4.2 are stale.) Update [verified 2026-08-10]: this pin has since diverged by build path. The upstream main pin is not the whole story — Luke Alonso’s b12x kernel library (the one eugr’s experimental fork image builds against) now requires nvidia-cutlass-dsl==4.6.0, while 4.5.0 is known to emit PTX that ptxas rejects on sm_121/sm_120 (NVIDIA/cutlass#3227, fixed in 4.5.1/4.5.2). So “4.4.2 is stale” is now itself stale: read the cutlass-dsl pin in the exact tag and build path you deploy.
Include nvcc if you rely on FlashInfer JIT; otherwise FlashInfer disables itself when prebuilt cubins are missing for your arch (§4.2).

10.3 The non-negotiable: validate on the image you deploy

Because Spark correctness hinges on whether the specific image carries sm_121 cubins and which backend auto lands on, always smoke-test before trusting a config, watching for both the hard error and silent garbage:

Serve one NVFP4 MoE model and one NVFP4-heavy dense model on the candidate image.
Watch logs for Failed to run cutlass FP4 gemm on sm120/sm121 … Error Internal or No compiled nvfp4 mm kernel for SM 121.
Inspect actual generated text for coherence (the silent-garbage failure mode produces no error).
Try, in order: (a) auto, (b) explicit --moe-backend flashinfer_b12x --linear-backend flashinfer-b12x, (c) forced Marlin. Adopt the fastest that is coherent and stable.
Fire a warm-up request (max_tokens=3, same model + chat template) at startup: the first request JIT-compiles Inductor/FlashInfer kernels (~25 s) and you don’t want a user to eat it.

10.4 Backend selection cheat-sheet for Spark

FP8 / BF16 model: auto is usually fine; FP8 uses native sm_121 tensor cores, so no Marlin dance in the common case. Treat it as image- and model-dependent, though: the v0.26.0 cu129 stable actually died on a CUTLASS FP8 W8A8 GEMM for our checkpoint (see the update timeline), so smoke-test the exact model and container before trusting auto.
NVFP4 model, want guaranteed-correct: --moe-backend marlin --linear-backend marlin.
NVFP4 model, want fastest native path: --moe-backend flashinfer_b12x --linear-backend flashinfer-b12x on v0.22.0+, then validate coherence.
Hybrid Mamba (nemotron_h): add --enforce-eager if your model/image combination shows the §8.3 graph-capture failure; it is a documented per-model workaround, not a Spark default.
Prefer CLI flags over the VLLM_* env vars: VLLM_NVFP4_GEMM_BACKEND was deprecated in v0.23 and removed in v0.25 (a no-op on current images), and VLLM_USE_FLASHINFER_MOE_FP4 is deprecated. Use --moe-backend / --linear-backend.

11. Appendix: glossary and file index

Glossary

GB10 / Grace Blackwell: the DGX Spark superchip (Grace ARM CPU + Blackwell GPU, unified memory).
sm_121 / CC 12.1: DGX Spark’s compute capability; workstation-Blackwell SM12x family, distinct from datacenter sm_100.
a vs f arch suffix: 12.1a = arch-specific cubin (one SM); 12.0f = family cubin (whole SM12x, needs CUDA 13).
CUTLASS: NVIDIA C++ template GEMM library; vLLM’s first-choice quantized matmul; needs arch-specific kernels.
FlashInfer: inference kernel library (attention, MoE, FP4/FP8 GEMM); ships prebuilt cubins or JITs at runtime.
CuteDSL: CUTLASS’s tensor-layout DSL; basis of the b12x SM12x kernels.
b12x: FlashInfer’s SM12x block-scaled fused-MoE / FP4-GEMM path; fast on Spark but opt-in (not auto, at the audited commit, §4.4).
Marlin / wna16: template-generated weight-only (W4/W8 ·A16) GEMM; broad arch coverage; Spark’s robust fallback for both dense and MoE.
QuTLASS: IST-DASLab CUTLASS extension providing FP4/MXFP4 kernels for sm_100/sm_120.
NVFP4: 4-bit float (e2m1), 16-element blocks, FP8-e4m3 block scales + FP32 global scale.
MXFP4: 4-bit float, 32-element blocks, e8m0 block scales; GPT-OSS format.
enforce-eager: disables CUDA-graph capture; needed for Mamba/hybrid on Spark; ~5–37% throughput cost.

File index (paths under the vLLM repo root)

Topic	Path
SM12x family guard	`csrc/cutlass_extensions/common.hpp` (`enable_sm120_family`)
CUTLASS FP8 dispatch (SM120)	`csrc/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8_dispatch.cuh`
CUTLASS FP4 entry + SM dispatch	`csrc/quantization/fp4/nvfp4_scaled_mm_entry.cu`
CUTLASS FP4 SM120 kernel	`csrc/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu`
Arch lists / family logic	`CMakeLists.txt` (`SCALED_MM_ARCHS`, `FP4_ARCHS`, `MARLIN_*_ARCHS`), `cmake/utils.cmake` (`cuda_archs_loose_intersection`)
FlashInfer wrapper / JIT	`vllm/utils/flashinfer.py`
`b12x` MoE expert	`vllm/model_executor/layers/fused_moe/experts/flashinfer_b12x_moe.py`
`b12x` linear kernel	`vllm/model_executor/kernels/linear/nvfp4/flashinfer.py`
Older FlashInfer CUTLASS MoE	`vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe.py`
NVFP4 MoE oracle (auto order, b12x exclusion)	`vllm/model_executor/layers/fused_moe/oracle/nvfp4.py`
NVFP4 linear oracle	`vllm/model_executor/kernels/linear/__init__.py`
MoE/linear backend options	`vllm/config/kernel.py`
Modular fused-MoE	`vllm/model_executor/layers/fused_moe/modular_kernel.py`
Marlin (dense)	`csrc/quantization/marlin/{marlin.cu,generate_kernels.py}`
Marlin MoE	`csrc/moe/marlin_moe_wna16/`
Marlin NVFP4 linear + scales	`vllm/model_executor/kernels/linear/nvfp4/marlin.py`, `vllm/.../quantization/utils/marlin_utils_fp4.py`
NVFP4 / ModelOpt quant method	`vllm/model_executor/layers/quantization/modelopt.py`
MXFP4 quant method	`vllm/model_executor/layers/quantization/mxfp4.py`
QuTLASS build	`cmake/external_projects/qutlass.cmake`
CUDA graph / eager config	`vllm/config/compilation.py`, `vllm/config/model.py`
Mamba graph hooks	`vllm/model_executor/models/{mamba,mamba2,jamba,nemotron_h}.py`
UMA / integrated-GPU detection	`vllm/utils/mem_utils.py`, `vllm/platforms/interface.py`
Install / aarch64 / Docker	`docs/getting_started/installation/gpu.cuda.inc.md`, `docker/Dockerfile`

Compiled from the vLLM main source tree (2026-06-02) and the official vLLM DGX Spark blog (2026-06-01). Backend selection details, especially the b12x opt-in, reflect current main; re-check the oracle comments in oracle/nvfp4.py against the exact image you deploy, since the upstream CUTLASS sm_121 MMA-guard fix may flip b12x into auto-selection in a future release.

vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes

Sun, 19 Jul 2026 00:00:00 GMT

Every performance number in this post was measured on our own two DGX Sparks and is labeled with the most precise identifiers we have for that run: image digest or pinned gitsha, checkpoint, and date. Where we cite someone else’s number, we say so. Where we were wrong earlier, the timeline says that too.

Why this post exists

The DGX Spark’s serving ecosystem moved faster than anyone’s documentation could; NVIDIA’s launch guidance and the vLLM team’s Spark deep-dive were each accurate for the moment they were written, and the platform kept moving underneath them. What that left for operators to discover in production: which image builds carry sm_121 kernels and which don’t, why some NVFP4 checkpoints crash or emit garbage on paths that work fine on datacenter Blackwell, what --gpu-memory-utilization really means on a unified 128 GB pool, and what happens to speculative decoding under concurrent load rather than in a batch-1 demo. We wrote our findings down after each incident. This post is that document, cleaned up, with the receipts, offered as a snapshot with dates attached, since it will age the same way.

Two boxes inform everything here: Spark-1 and Spark-2, both GB10 (aarch64, sm_121a, 128 GB unified, ~273 GB/s), serving production traffic for a homelab agent fleet (reasoning, tool-calling, and structured extraction), 24/7, behind a model gateway. This is not a benchmark rig; every lesson below came out of something breaking. For the kernel-level why behind these notes (the CUTLASS/FlashInfer/Marlin mechanics, the backend oracle, and the arch details), see the companion technical report.

The five-minute model of what vLLM is doing on this box

Everything in this post follows from four pieces of vLLM machinery. If you already operate vLLM, skip ahead; if not, these four paragraphs are the vocabulary the rest of the post uses.

1. Serving has two phases with opposite bottlenecks. Prefill processes your whole prompt in parallel; it’s compute-bound, measured in thousands of tokens/second, and determines time-to-first-token. Decode generates one token at a time, and each token requires re-reading the model’s active weights from memory; it’s memory-bandwidth-bound, which on a 273 GB/s unified-memory box is the budget everything competes for. This split explains most of the numbers here: why a 35B MoE with only 3B active parameters decodes fast on a Spark (decode reads the active set, not the total), why a weight-only quantization like Marlin costs little at single-user decode but shows up at prefill and concurrency (those are compute-heavy), and why our tables always report the phases separately. Two caveats the rest of the post leans on. First, the short-context limit: decode actually re-reads the active weights and the KV cache of every token attended so far, so at long context the KV term grows and decode throughput falls with sequence length, which is why KV-pool sizing (§9) is a performance concern, not just a capacity one. Second, what Marlin means throughout: it’s vLLM’s weight-only 4-bit kernel that keeps weights 4-bit in memory and dequantizes them to BF16/FP16 inside the GEMM, so it wins on bandwidth-bound decode but runs the matmul in high precision, never touching the native FP4 tensor cores. That’s why its cost surfaces in compute-bound prefill and under concurrency, not at batch-1 decode.

2. Kernels are chosen by a chain with three links, and any link can break. When vLLM loads a model it reads the checkpoint’s quantization config (config.json), runs it through a backend “oracle” (select_nvfp4_moe_backend() in vllm/model_executor/layers/fused_moe/oracle/nvfp4.py) that maps quant format + GPU architecture to a ranked list of candidate kernel backends. That ranked list is the oracle’s AVAILABLE_BACKENDS constant: the ['FLASHINFER_TRTLLM', … 'FLASHINFER_CUTLASS', … 'MARLIN', 'EMULATION'] line you’ll see in our boot logs. The winning backend then still has to find a compiled kernel binary (cubin) for your GPU inside the wheel you actually installed. Three links: checkpoint declares, oracle selects, wheel delivers. Each fails differently: a W4A16 checkpoint means the oracle correctly selects Marlin (nothing to fix); a wheel without sm_121 cubins means the selected backend dies at runtime (no kernel image is available); and a version-gated integration mismatch can silently drop you onto slow fallbacks. Most of this post’s timeline is one of those three links breaking.

3. The KV cache is a pre-allocated pool, not a meter. vLLM (PagedAttention) grabs one big block of memory at startup, sized by --gpu-memory-utilization, and pages sequences into it. It’s capacity; the boot log tells you the pool size in GiB and in tokens, and dividing them gives the per-token cost that makes context-length planning arithmetic instead of folklore (worked example in the unified-memory section). On a unified-memory box the grab itself is the danger: it can’t be swapped, so an oversized pool is the OOM.

4. Speculative decoding buys speed with a coupled scheduler. A cheap drafter proposes k tokens; the full model verifies them in a single forward pass; accepted tokens cost almost nothing extra. Acceptance depends entirely on how predictable your output is to that drafter (code and structured JSON accept well, free prose doesn’t) and the metric vLLM reports (“acceptance length”) includes a guaranteed bonus token, so read it carefully. The part nobody advertises: the drafter is wired into the scheduler, so turning speculation on can change scheduling behavior itself: disable async scheduling, cap the token budget per step. A “free” flag can therefore tax prefill and TTFT even while decode acceptance looks healthy. That coupling is exactly how our n-gram experiment lost (full numbers below).

The timeline: from launch-day optimism to a config that survives

This is the part most guides skip: when each fact became true. sm_121 support moved so fast that advice written eight weeks apart contradicts itself, including our own. Dates below come from our git history, incident notes, and the vLLM release record.

2025-12-20: the origin. NVIDIA’s launch-window vLLM container documentation (NGC vllm:25.11-py3 and its release notes) claimed clean “DGX Spark functional support” and NVFP4-on-Blackwell. We wrote our first Spark serving doc from those sources, taking the claims at face value. What wasn’t yet visible in that material: sm_121 is consumer Blackwell, not the sm_100 the FP4 kernel work had targeted, and none of the then-shipped FlashInfer wheels contained sm_121 cubins, details the launch-window docs preceded rather than concealed. Everything below is the gap between documentation written at launch and a working box, closing in real time.

2026-02 → 2026-05: upstream catches up, in pieces. The vLLM release record tells the real support story: enable_sm120_family (#33517, Feb); “Fix DGX Spark logic” (#38126, → v0.19.0); disable the broken FlashInfer CUTLASS MoE on sm_121 as a stopgap (#39825, → v0.20.0); fix the wrong sm_121 exclusion from Marlin/CUTLASS FP8 (#35568) and land the b12x sm_121 kernels (#40082), both only in v0.22.0 (2026-05-29). In our tested pre-v0.22 images and checkpoints, native NVFP4 paths crashed (cutlass FP4 gemm ... sm120/sm121 ... Error Internal), silently emitted garbage, or required Marlin; behavior varies with the image build, packaged wheels, checkpoint, and backend, so treat this as our observed floor, not a universal law.

2026-05-31: the empirical break. After weeks of intermittent failures we isolated the pattern on our own boxes: prebuilt FlashInfer wheels shipped zero sm_120/121 cubins; the model-card env vars rerouted only MoE layers while dense NVFP4 linears still hit the broken CUTLASS path; and the reliable fix was forcing Marlin for both backends (plus --enforce-eager for hybrid-Mamba models, a graph-capture bug fixed later in v0.22.0). Forced-Marlin became our fleet default, correct at the time, and understandably carried forward by community guides (including ours) as if it were a permanent hardware rule.

2026-06-01/02: reconciliation, then past Marlin. The vLLM team published its Spark deep-dive claiming auto backends were fine, which contradicted our validated config. Reconciling blog-vs-repo-vs-our-logs showed the disagreement was a version gap, not a dispute: our Marlin era ran on v0.20.0, which simply lacked the v0.22.0 kernels. On 2026-06-02 we moved to a digest-pinned cu130 nightly and --moe-backend auto selected FLASHINFER_CUTLASS native FP4, coherent, no Marlin, no enforce-eager, and faster (~50.2 → ~55.8/58.5 tok/s on our Nemotrons). Meanwhile stock v0.22.0-cu129 (same vLLM version, different wheel) still crashed on auto because the cu129 wheel didn’t bake the sm_121 MoE cubin. That’s when “which vLLM version?” stopped being the right question and “does this wheel carry the cubins?” replaced it.

2026-06-05 → 2026-07-05: the MTP production saga. We deployed the 35B Qwen with in-checkpoint MTP: 86.8% acceptance in validation, then CUDA error: an illegal memory access crashes every few hours in production, only under concurrent load (upstream #41190/#42084; the failure signature is a broken draft path poisoning the accepted-token buffer). An image bump to a 0.23-era nightly fixed the load-time bug (#35031 via #46316) but not the runtime crash, so spec-decode went off. Along the way, a second, unrelated checkpoint lesson: the W4A16 build’s 4-bit lm_head produced !!!! garbage on greedy structured-extraction output; quantization choices inside the checkpoint, not the engine, decided both failures.

2026-07-11/12: the W4A4 resolution. Swapping to a genuine W4A4 checkpoint (4-bit activations, lm_head unquantized) moved experts and drafter onto the FLASHINFER_CUTLASS path, and the crash was gone: 8-way soak, 400/400, zero IMA, then re-enabled in production and bumped to num_speculative_tokens: 3 (mean acceptance length ~3.1 output tokens per verification step, bonus token included; ~95% draft acceptance on real traffic). The upstream bugs remain open; we didn’t patch vLLM, we routed around the crashing kernel path via configuration, checkpoint format and drafter backend together, so neither alone gets the credit. The transferable lesson stands regardless: on sm_121, the checkpoint’s quant format (W4A4 vs W4A16) decides which kernels you run, and therefore which bugs you hit.

2026-07-19: today. Both boxes on digest-pinned nightlies (0.22/0.23-era), native FP4, MTP where the model ships heads, everything below measured on this config. We smoke-tested the v0.25.1 stable candidates the same day this post was assembled, and the timeline’s lesson reproduced on cue: the official cu129 stable still ships no sm_121 cubins (dead at engine init), and the promising new cu13 tag turned out to be a mis-stamped dev artifact decoding 2.6× slow (details in the image table below). Seven months in, the rule hasn’t changed: there is still no viable aarch64 stable for GB10 native FP4; validate the wheel you actually pulled, then pin it.

Update: 2026-08-05 (post-publication). Three developments the timeline’s own logic predicted: (1) v0.26.0 (2026-07-27): we smoke-tested its cu129 aarch64 stable on a scratch GB10 instance, and the sm_121 NVFP4 cubins are finally present. auto selects FLASHINFER_CUTLASS for both dense and MoE NVFP4 with no no kernel image is available, breaking the three-release cu129 drought. It still could not serve our unsloth checkpoint, though: EngineCore init dies during the profiling run on a CUTLASS FP8 W8A8 GEMM (cutlass_gemm_caller.cuh:62, Error Internal), a path the same checkpoint clears on our v0.23-era nightly. So the operator bottom line is unchanged (no cu129 stable serves this GB10 NVFP4+FP8 workload yet), but the failure moved from a missing NVFP4 kernel image to an FP8 CUTLASS error, worth re-testing per checkpoint. (Scratch-tested 2026-08-05; not yet root-caused or filed.) (2) #41190 (the MTP illegal-memory-access) was fixed on main 2026-07-29 (#48245/#49620), though not yet in a tagged stable release and marked a probable (“may fix”) rather than confirmed resolution, so stable-wheel operators remain exposed until a nightly carries it. (3) Prefix caching + MTP now coexist on a v0.26.1-dev nightly: the either/or that #42084 forced looks resolved via an experimental Mamba/SSM cache-align mode (tested on our GDN-hybrid 35B: zero out-of-bounds under a shared-prefix + MTP soak). All are elaborated in the sections below; noted here to keep the timeline honest about its own aging.

The one rule that explains all the FP4 confusion

A widely-shared rule (reasonably derived from what its authors tested) says “GB10 has no native FP4 compute, so FP4 MoE must run on Marlin.” Our production logs show the rule has an important exception:

Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends:
  ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED',
   'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION']

That’s a Qwen3.6-35B-A3B NVFP4 MoE on --moe-backend auto, coherent output, running for days under real load, on a GB10. No Marlin anywhere.

Native FP4 on sm_121 needs two conditions, and both must hold:

The image’s FlashInfer wheel actually bakes sm_121 cubins. Source support is not wheel support. Stock v0.22.0-cu129 has the sm_121 kernels in source (#40082) but the cu129 wheel ships without the MoE cubin; auto dies with no kernel image is available. The cu130 nightlies bake it. This is an image property; smoke-test the exact digest you deploy.
The checkpoint is genuine W4A4 (nvfp4-pack-quantized: 4-bit weights and activations). A W4A16 weight-only “NVFP4” checkpoint has no FP4 activations, so there is no FP4 matmul to dispatch; auto lands on Marlin correctly, and no flag will change it.

Condition 2 is the one nobody talks about, and it explains the contradictory community reports. We ran three quantizations of the same model (Qwen3.6-35B-A3B) on the same image:

Checkpoint	Format	What `auto` picks
`nvidia/Qwen3.6-35B-A3B-NVFP4` (modelopt)	W4A16 weight-only	MARLIN
RedHatAI NVFP4 build	W4A4	FLASHINFER_CUTLASS
`unsloth/Qwen3.6-35B-A3B-NVFP4` (deployed)	W4A4	FLASHINFER_CUTLASS

So when a guide says Marlin is required, it’s worth checking which checkpoint was tested; most popular recipes use the W4A16 builds, which made the conclusion easy to reach. If it was an nvidia/... modelopt W4A16 build (most of the popular recipes are), their observation was correct for that checkpoint; it just doesn’t generalize into a hardware rule.

Decision tree:

Is the checkpoint W4A4 (nvfp4-pack-quantized, activations num_bits: 4)?
├─ NO (W4A16 weight-only) → auto → Marlin. This is correct; don't fight it.
│    At batch-1 you barely pay for it: decode is bandwidth-bound and 4-bit
│    weight storage already captures the win. Marlin costs you prefill + concurrency.
└─ YES → does your image's flashinfer wheel bake sm_121 cubins?
     ├─ NO (e.g. stock v0.22.0-cu129) → force --moe-backend marlin, or change image
     └─ YES (cu130 nightlies, pinned by gitsha) → --moe-backend auto
          → FLASHINFER_CUTLASS native FP4, dense AND MoE. Faster than Marlin,
            full cudagraphs, and (0.23-era) --attention-backend flashinfer
            works even on hybrid attention/Mamba/GDN models.

One more wrinkle from the release archaeology: the fastest sm_121 path, flashinfer_b12x (#40082), is intentionally excluded from auto-selection pending an upstream CUTLASS MMA-guard fix, so auto gives you a working-and-fast path, not necessarily the theoretical-fastest one. Precision note, since people assume this is MoE-only: it isn’t. The dense NVFP4 linear oracle (_POSSIBLE_NVFP4_KERNELS in vllm/model_executor/kernels/linear/__init__.py) comments b12x out of its auto candidate list with the same rationale as the MoE oracle, so auto never even tests it on either path. Our boot log’s FlashInferCutlassNvFp4LinearKernel for the dense GEMM is auto walking past the policy-excluded b12x (and a higher-priority CuteDSL kernel our wheel can’t use) down to CUTLASS. Both dense and MoE land on CUTLASS for the same reason: b12x is opt-in only (--linear-backend flashinfer_b12x / moe-backend flashinfer_b12x), never auto-selected. We haven’t needed to force b12x; CUTLASS is already ahead of Marlin.

Which images actually work (July 2026)

The single most common failure mode on Spark is an image whose wheels predate sm_121. Verified state as of 2026-07-19 (Docker Hub API + our deployments):

Image	vLLM	sm_121 verdict
`vllm/vllm-openai:nightly-<gitsha>` (cu130 line)	0.23.x-dev	✅ our Spark-2 production: native FP4 + MTP, pinned by gitsha
`vllm/vllm-openai:nightly-4721bb3aa…`	0.22.1rc1.dev26	✅ our Spark-1 production (Nemotron ×2, native FP4)
`vllm/vllm-openai:v0.25.1-aarch64-cu129-ubuntu2404` (stable, 2026-07-13)	0.25.1	❌ tested 2026-07-19: dead at EngineCore init, `CUDA error: no kernel image is available` in the FlashInfer alloc path. cu129 stable wheels have now shipped without sm_121 cubins for three consecutive releases (0.22.0 → 0.25.1)
`vllm/vllm-openai:vllm-arm64-cu13-0.25.1-<sha>` (2026-07-17)	says 0.25.1…	❌ tested 2026-07-19: a trap. Looks like the long-awaited cu13 aarch64 stable, and the sm_121 cubins ARE there (`auto` → FLASHINFER_CUTLASS, coherent output), but `/version` reports `0.1.dev18498`, a mis-stamped CI artifact, and decode ran 2.6× slower than our pinned 0.22-era nightly on the identical config (23.5 vs 62.0 tok/s, prefill halved too)
`vllm/vllm-openai:cu130-nightly`	~0.20-era	❌ dormant since 2026-04-23: it still appears in guides written before the track went quiet (easy to miss; we only caught it via the Docker Hub API); the cu13 line moved to `nightly-dev-arm64-cu13.0.1-<sha>`
`v0.22.0-aarch64-cu129` + forced Marlin	0.22.0	✅ as a pinned stable fallback only (weight-only path, ~50 tok/s on our 30B Nemotron)

Rules that survived contact:

Check /version before anything else: a tag’s name is a claim, not a fact. The cu13 “0.25.1” tag above passed every kernel check and would have shipped a 2.6× decode regression; the one-line curl :PORT/version exposed it as a dev artifact instantly. Our best hypothesis for the slowdown: with a bogus version stamp, version-gated integrations (FlashInfer wheel matching, autotune caches, compat shims) can’t resolve, and the build silently takes fallback kernel paths; the uniform ~2× hit across both prefill and decode fits systemic fallback, not one bad kernel. We didn’t profile further; a mis-stamped build is disqualified either way.
Pin nightly-<gitsha> or a digest, never a moving tag. We froze our production image as a locally-tagged derivative so upstream tag drift can’t change what’s running.
Smoke-test the exact image before deploying: boot the model on a scratch port, hit the raw /v1/completions endpoint (bypasses chat template and reasoning parser), and watch for both the loud failure (no kernel image is available, cutlass FP4 gemm … Error Internal) and the quiet one (silent garbage tokens).
A reasoning model returning empty output is usually not broken. If finish_reason: length and both content and reasoning_content are empty, the model hit max_tokens inside an unclosed <think> block. Raise the budget or pass chat_template_kwargs: {"enable_thinking": false} before you blame the kernels.

The MTP saga: speculative decoding that survives real traffic

Qwen3.6 ships in-checkpoint MTP heads, and single-stream decode on a bandwidth-bound box is exactly where speculative decoding pays. Getting it to stay up taught us the most expensive lesson in this post:

Batch-1 validation lies. Our first MTP deployment (W4A16 checkpoint → Marlin experts, drafter forced to a Triton MoE backend) validated clean at batch-1 with 86.8% acceptance, then died in production with CUDA error: an illegal memory access only under concurrent load, taking both the coordinator tool-calling endpoint and the memory-extraction alias down with it. Upstream issues existed (#41190/#42084); an image bump didn’t fix it; we ran months with spec-decode off.

The crash was avoided by configuration, not a patch. Moving to the W4A4 build routed both the experts and the drafter through FLASHINFER_CUTLASS, a different kernel path than the crashing Marlin+Triton combination. The W4A4/CUTLASS configuration eliminated the crash in our concurrency soak, but checkpoint format and drafter backend changed together, so the result cannot be attributed to either change alone. Before re-enabling in production we soaked it at 8-way concurrency: 400/400 requests, zero illegal-memory-access, 73% acceptance under load.

What MTP actually delivers on GB10 (our production numbers, num_speculative_tokens: 3):

Mean acceptance length ≈3.1 output tokens per verification step under 8-way burst: note vLLM’s metric includes the guaranteed bonus token, so with n=3 that’s ~2.1 accepted draft tokens per step (per-position draft acceptance 84.2% / 67.8% / 57.5%: the falling marginal value is why you keep n at 1–3, unlike block-verify methods that run 7–15).
~94.8% cumulative acceptance over ~20 h of real coordinator + extraction traffic, far above any synthetic number, because real agent workloads are predictable, structured output. Measure acceptance on your traffic via /metrics | grep -i spec_decode, not on a demo prompt.
Thinking-mode decode is faster than answer-mode with MTP on: structured reasoning accepts better than free prose.

Measured: native FP4 + MTP vs the published Marlin numbers

We ran the same single-stream methodology the community playbook published (prefix-cache-defeating nonces, forced full-length generation, server-side token counts, 3-run medians, ~6K-token prompt) against our production 35B, native FLASHINFER_CUTLASS, W4A4, MTP-3, vLLM 0.23.1rc1-era, measured 2026-07-19:

Config (same model, Qwen3.6-35B-A3B NVFP4)	TTFT	Prefill	Decode (answer)	Decode (thinking)
Ours: W4A4 → native FP4, MTP-3, live prod box	955 ms	6,307 t/s	82.2 t/s (79–98 across runs)	102.5 t/s
Published: W4A16 → Marlin, MTP-3, idle box (vlaicu.io)	~965 ms	~6,240 t/s	102 t/s	117.4 t/s (6K ctx)
Ours, historical: W4A16 → Marlin, MTP-3 (2026-06-06)	~91 ms @ short prompt	N/A	~102.7 t/s	N/A

The honest read: these measurements show no batch-1 native-FP4 decode advantage, and they don’t isolate the backend, either. Checkpoint format, machine load, measurement date, and tooling all differ across the rows, so this is convergent evidence, not a controlled experiment. It is consistent with batch-1 parity: decode is memory-bandwidth-bound and 4-bit weight storage is identical either way; our own W4A16-era measurement (102.7 t/s) matches the published Marlin number almost exactly, and our W4A4 medians land lower mostly because the box was serving background production traffic during the run (top of spread 97.6; window acceptance 76.4% vs ~80% idle). Calling it backend equivalence would need a same-checkpoint, same-load A/B we haven’t run. Native FP4’s real wins are prefill/concurrency (compute-bound paths), full cudagraphs on the hybrid arch, and (decisively for us) the kernel path on which MTP stopped crashing. The practical takeaway: don’t switch backends expecting a batch-1 decode win, and when evaluating Marlin, look at prefill and concurrent load: that’s where the difference actually lives.

Patterns from this configuration (one model, one serving config, corroborated where noted, not universal laws): prefill held ~6.3K t/s in both modes on this Qwen3.6-35B setup (TTFT ≈ prompt_tokens ÷ 6300; the vlaicu measurement of the same model reports the same ~6.2K, and note our Nemotron prefills at 8.5K; ‘flat across modes’ generalizes, the absolute number doesn’t); thinking decoded ~25% faster than answering under MTP for this workload and acceptance profile (102.5 vs 82.2); per-position draft acceptance decayed 86.6% → 77.2% → 65.5% across positions 0/1/2 in the same window.

And the no-speculator baseline, same script, same day, our Nemotron-3-Nano-30B-A3B-NVFP4 (native FP4, no spec-decode, vLLM 0.22.1rc1-era):

Metric	Answer-only	Thinking
TTFT (~6K prompt)	706 ms	710 ms
Prefill	8,530 t/s	8,485 t/s
Decode	64.2 t/s	63.8 t/s

Two things worth noticing: thinking ≈ answering exactly as bandwidth-bound theory predicts when there’s no draft head (the Qwen thinking-speedup is entirely MTP), and the Mamba-hybrid prefills ~35% faster than the attention-hybrid Qwen (8.5K vs 6.3K t/s). Also 64.2 t/s is ~10% above our own 2026-06-02 measurement of the same deployment (58.5), measured with a different tool; methodology moves numbers, which is exactly why every figure here states its method and date.

Two spec-decode paths we evaluated but do not run:

n-gram lookup: the best story in this post, because we got it wrong in public first. Our initial A/B showed n-gram losing 14% on our extraction workload despite 62% acceptance, and we wrote it up as “the free flag isn’t free.” A reviewer pushed back: our explanation blended three effects, and the prefill penalty looked configuration-specific. The controlled retest (every arm with an explicit --max-num-batched-tokens, prefix caching pinned off, k swept 1→4, token-equality checks) flipped the verdict entirely: extract-workload decode went 61.5 → 117.3 tok/s (+91%) at k=3, with no prefill penalty in any arm. The original loss was our configuration all along (an unset scheduler budget interacting with speculation settings), exactly what the boot warning had been trying to tell us. The k-sweep shows the real economics: acceptance decays with depth (80.8% at k=1 → 64% at k=4) while throughput saturates at k=3. Two caveats survived the retest: speculative and plain decode are not token-identical at temperature 0 on this build (every spec arm diverged from the control’s greedy output at the same early position: deterministic batch-shape numerics, so validate output quality, don’t assume equivalence), and all of this is single-stream: speculation taxes throughput under concurrent load, so it ships with --speculative-disable-by-batch-size or not at all. The meta-lesson outranks the numbers: a configuration confound can invert your benchmark’s conclusion, and the warning that would have told you is in the boot log.

And then the story turned again. Before promoting, we ran the production gates (an 8-way soak plus an output-quality gate with a control arm), and the quality gate killed it: the no-spec control produced 20/20 perfect JSON extractions, while ngram produced 0/20, with version strings losing their final component ("0.22" for "0.22.0") and JSON keys mutating mid-document ("sentence" → "sent"). The GPU proposer failed identically, localizing the defect to the shared verification path. The temperature-0 divergence we’d politely called “batch-shape numerics” was token corruption all along. (Two more gate catches on the way: prefix caching × ngram hard-asserts in the Mamba-2 layers on this arch, and the burst-protection flag most guides recommend doesn’t exist on this build in either spelling.) Final ledger for n-gram on this box: +91% throughput, mechanically real, generating subtly wrong tokens, worthless on this hybrid Mamba/GDN architecture (the corruption lives in the shared verification path, upstream #39273; CPU and GPU proposers fail identically, which localizes it there, and on a pure-attention target n-gram may well be fine). Three verdicts in three days, each overturned by a better-controlled experiment than the last. If there’s one sentence to take from this post, it’s the last gate’s: speculative-decoding throughput without output validation is not a result.
DFlash: block-diffusion drafters, the biggest published single-stream wins (2.2–2.7× on GB10, per AEON-7’s measurements). The caveat that keeps it off our fleet: on quantized (NVFP4/FP8) targets under stock vLLM, acceptance collapses (~4 of 15 tokens → a net slowdown); the working paths need a patched sm121 build or a BF16 target, and DFlash requires BF16 KV, incompatible with our fp8-KV recipes. MTP at ~95% real-traffic acceptance with zero extra memory sets a high bar.

Unified memory: the OOM you get with “free” RAM showing

--gpu-memory-utilization on Spark is a fraction of the whole 128 GB pool, shared with the OS, page cache, and every other container. Two facts make it dangerous:

The reservation is unswappable. On unified memory, vLLM’s KV-pool grab can’t be paged out. Set 0.6 on a box that’s also running a gateway and observability stack and you’ve made the kernel OOM-killer’s decision for it: that exact misconfiguration cost us a 2h40m production outage (instance reserved ~71 GiB, host hit ~2 GiB free with swap exhausted, oom_kill).
Lowering it shrinks KV capacity, not speed. The pool is capacity, not a performance knob. Our 35B runs at 0.3 (≈34 GiB) while keeping the model’s full 262K context, because the math works:

KV cost  ≈ 12.4 KB/token          ← boot log: 7.15 GiB ÷ 617,494 pool tokens
                                     (Qwen3.6-35B-A3B, KV cache in fp8)
1 × 262K sequence ≈ 3.1 GiB
0.3 util → 7.15 GiB KV pool → 2.36x concurrency at full 262K context
                                     (vLLM prints this: Maximum concurrency 2.36x)
real coordinator/extraction traffic touches < 2% of that

A clarification that trips people up: two quantizations coexist in this server and they are different knobs. The weights are NVFP4 (that’s the checkpoint format the whole kernel-selection story is about); the KV cache is fp8 because we pass --kv-cache-dtype fp8. The 12.4 KB/token figure is a property of the KV dtype and the model’s attention layout; the weight quantization plays no part in it. (A widely-quoted figure for this same model family is ~16.6 KB/token, measured on a different checkpoint, vLLM build, and config. Hybrid attention/SSM models are exactly where per-token KV accounting varies across versions, another reason the formula beats the folklore: your own boot log prints both numbers; divide them.)

Read Available KV cache memory and GPU KV cache size ... tokens from your own boot log, and size gpu-memory-utilization from max-num-seqs × max-model-len plus slack, not from a recipe’s 0.85 default, which assumes the box serves nothing else. If a load fails with memory apparently free, NVIDIA’s documented debugging workaround for UMA memory-reporting is to flush the page cache (sync; echo 3 > /proc/sys/vm/drop_caches) and then restart the affected application; the cache holds unified memory CUDA can’t reclaim. Treat it as a diagnostic step, not routine ops.

Smaller sharp edges, quickly

Tool calling fails closed, loudly, and only at request time. Without --enable-auto-tool-choice --tool-call-parser <family>, the model loads fine and /v1/models is green; then every agent request 400s ("auto" tool choice requires…). The parser tracks the model family, not your client: qwen3_coder/qwen3_xml for Qwen3.6, nemotron_v3/nano parsers for Nemotron. The wrong parser fails open: the call text sits in content, tool_calls stays null, and your agent silently stalls after the think block.
Hybrid-Mamba graph capture (nemotron_h): needed --enforce-eager on v0.20.0 (~37% throughput tax), fixed by v0.22.0; drop the flag and take the free speed.
max_tokens starvation on reasoning models breaks tool-call parsing in ways that look like model bugs. Budget ~10K+ output tokens for reasoning models in agent pipelines. It presents identically to the wrong-parser stall above (the agent hangs after the think block with no error), but the two are distinguishable: wrong parser gives tool_calls null with the call text sitting in content; budget starvation gives finish_reason: length with an unclosed <think> block. Check which before blaming the model.
ConnectX-7 clustering: the 200 Gb/s per QSFP cable arrives as two ~100 Gb/s interfaces: configure both or you get half. Each port exposes two RoCE “twins” (each behind an x4 PCIe Gen5 link); our single-interface ib_write_bw measured ~111 Gb/s, consistent with one twin, so full NCCL bandwidth needs NCCL_IB_HCA=<twin1>,<twin2>. DGX Spark also doesn’t support GPUDirect RDMA; CUDA-memory traffic takes NVIDIA’s documented host-pinned fallback. And multi-node vLLM needs more than NCCL: a Ray cluster (or multi-node multiprocessing) with matching runtime environments and model paths, plus an explicitly selected distributed executor. TP=2 across two one-GPU Sparks works; PP=2 can reduce cross-node tensor-parallel collectives where the model supports it.
Cold start is real: ~2.5 min weight load + ~25 s first-request JIT on our 30B-class models. Keep instances warm (no idle-timeout, no on-demand start in the request path) and fire a max_tokens: 3 warm-up ping at boot.

What we’d tell you to deploy today

For a 30–35B-class MoE NVFP4 reasoning model on one GB10, July 2026:

Checkpoint: a genuine W4A4 build (check config.json for nvfp4-pack-quantized with activation num_bits: 4; confirm lm_head is in the quant ignore-list if you do structured extraction; 4-bit lm_head gave us !!!! garbage on greedy JSON output).
Image: a cu130-line nightly pinned by gitsha (ours: 0.23.1rc1-era), or v0.25.1 stable after an sm_121 smoke test.
Flags: --moe-backend auto (verify the boot log says FLASHINFER_CUTLASS), --kv-cache-dtype fp8, MTP at num_speculative_tokens: 3 after an 8-way soak, tool-call + reasoning parsers matched to the model family, and --gpu-memory-utilization computed from the KV math above, not defaulted.
Process: pin digests, smoke-test raw completions on every image change, watch /metrics spec-decode acceptance and KV utilization, and never validate spec-decode at batch-1.

Still open: what we’re waiting on, and why it matters

A snapshot of the upstream gaps that shape our config today. Each of these is a reason some recommendation above reads “pinned nightly” or “re-test later” instead of “just do X.”

vLLM #41190: illegal memory access in MTP speculative decoding (fixed on main 2026-07-29 via #48245/#49620; not yet in a tagged stable release, and upstream marks it a probable “may fix” rather than a confirmed resolution). The crash that took our 35B down for a month. Failure signature: the drafter path poisons the accepted-token buffer (all -1) and the engine dies with CUDA error: an illegal memory access, only under concurrent load, which is why batch-1 validation sails past it. The upstream fix locates the root cause deeper than our symptom: async-decode output-placeholder accounting underflows when KV-pressure preemption rolls back an in-flight request. That’s a scheduler bug, kernel-agnostic, which is precisely why concurrency (KV pressure, then preemption) triggers it and batch-1 never can. Why it’s pertinent: we didn’t patch it, we routed around it. The W4A4/CUTLASS configuration avoids the crashing combination, but since the true cause is scheduler-level, we can’t yet say whether we avoided or merely masked it. We ran that soak: W4A4/CUTLASS held at 100% KV utilization under concurrent MTP load with zero illegal-memory-access, but with prefix caching off, vLLM’s admission control never actually preempted an in-flight request, so saturation alone did not reproduce the crash. That is strong evidence the config is robust in practice, and a hint that the #41190 rollback needs conditions this config avoids, but it is not a formal proof of avoided over masked. Anyone on a W4A16 NVFP4 checkpoint (the Marlin path) with MTP under real concurrency stayed exposed until a nightly carries #48245, one more reason to run digest-nightlies, not stables. Our rule stands regardless: no spec-decode config goes to production without an 8-way soak.
vLLM #42084: out-of-bounds gather with prefix caching + spec decode on sm_121a (open, but see the 2026-08-05 update below). Constrains which feature combinations are safe on this architecture. The mechanism is hybrid-architecture-specific: the linear-attention state block table (Mamba-2 layers in Nemotron, Gated DeltaNet layers in Qwen3.6, both managed by vLLM’s SSM cache, which it confusingly calls “Mamba” even for GDN) and the speculative extension disagree on how many blocks a sequence owns, so the torch.gather indexes past the end; a pure-attention model has no second state block table to desync, so it is likely unaffected. Pertinent because prefix caching and speculation are both individually attractive for agent workloads: this is the reason we validate them together, not separately, before enabling both. (We did lose the production 35B to an illegal-memory-access around 2026-08-04 under concurrent extraction load, but with no logs and a coincident llamactl-restart incident on the same box we can’t attribute it — not to #42084, and not confidently to any particular feature combination — so we don’t count it as evidence either way.) Update 2026-08-05: on a v0.26.1-dev nightly we ran prefix caching + MTP together on our GB10 Qwen3.6-35B (the Gated DeltaNet hybrid, not classic Mamba) under a shared-prefix concurrent soak (40% cache hit, 86.9% KV): zero out-of-bounds, zero state-cache asserts, coherent output throughout. #42084 appears resolved via a new experimental cache-“align” mode that sets the attention block size to match the linear-attention page size. So the prefix-caching-vs-spec-decode either/or looks liftable on recent nightlies, with the caveat that vLLM still marks the align path experimental.
The b12x backend is deliberately excluded from auto selection pending an upstream CUTLASS sm_121 MMA-guard fix (the exclusion is a documented choice in vLLM’s NVFP4 oracle). b12x (#40082) is the purpose-built sm_121 path and plausibly the fastest one, meaning some performance is intentionally left on the table today. When that guard fix lands, auto may silently start selecting b12x, which is both good news and a behavior change to catch in the boot log. We re-bench on every image bump partly for this.
No aarch64 stable wheel has shipped sm_121 cubins in three consecutive releases (v0.22.0 → v0.25.1, verified by us on the day each mattered), and the first CUDA-13 aarch64 “stable” tag turned out to be a mis-stamped dev artifact. This is the single reason our production images are digest-pinned nightlies, not a preference for living dangerously, but the only track whose wheels currently carry the kernels. The day a properly version-stamped cu13 aarch64 stable appears (check /version first), the calculus flips.
NGC container lag (vllm #31424). NVIDIA’s official vLLM containers trail upstream by months: the newest NGC tag ships 0.22.1 while upstream stable is 0.25.1. Pertinent because “use the vendor container” is the default advice in most enterprise settings, and on this platform it currently can’t serve the newest model architectures or fixes. (NVIDIA closed #31424 as not_planned, so the lag is effectively permanent-by-neglect, which is why the vendor container can’t be the default on this platform.)
Speculative decoding’s scheduler coupling: on this build, CPU n-gram speculation disables async scheduling, and the per-step token budget derives from your batch/speculation settings (vLLM says both in boot warnings). Mostly mitigable from your side: our controlled retest showed that simply setting --max-num-batched-tokens explicitly eliminated the penalty that sank our first experiment, and ngram_gpu (present and working on our 0.22-era build) didn’t disable async scheduling at all. What remains open upstream, now upgraded from observation to confirmed defect: n-gram speculation on this build corrupts structured output (control 20/20 clean vs ngram 0/20; dropped token components inside version strings, mutated JSON keys; identical signature from CPU and GPU proposers → shared verify-path bug), and prefix caching × ngram hard-asserts in mamba_mixer2 on the hybrid arch. Before filing, we duplicate-checked the tracker, and both bugs were already known, which is its own lesson: the assert is precisely open PR vllm#46424 (our data extends it from MTP to ngram), and the corruption belongs to the vllm#39273 hybrid-corruption cluster. We also tested vllm#40875’s config-only workaround (prompt_lookup_min=8): it did not mitigate on this Mamba2 hybrid: corruption changed shape but persisted, 0/20. Our contribution upstream is corroborating data on existing issues, not new tickets, and until that cluster resolves, no n-gram config passes our quality gate regardless of its throughput.
One we owe upstream: in our n-gram A/B, temperature-0 output lengths diverged run-to-run with speculation on (502–1116 tokens) while the control was deterministic (529 every run). Speculative decoding is supposed to preserve the target distribution; numerics on this path apparently aren’t bit-stable on this build. We haven’t filed a reproducer yet; until then treat it as our observation, not a confirmed upstream bug.

Appendix: every knob this post mentions

Flag	What it actually controls	If you get it wrong
`--gpu-memory-utilization`	Fraction of the (unified!) pool pre-allocated for weights + KV	Too high on a shared box = unswappable reservation = OOM-killer (our 2h40m outage)
`--max-model-len`	Max prompt+output tokens per request; sizes one sequence’s KV claim	Model-card maxima can reserve absurd KV on 128 GB: compute from the KV math
`--max-num-seqs`	Concurrent sequences scheduled	High values tax bandwidth-bound decode; also interacts with cudagraph capture sizes
`--quantization`	Overrides quant-format detection (`modelopt` for nvidia ModelOpt checkpoints)	Compressed-tensors checkpoints auto-detect; passing the flag wrongly breaks loading
`--moe-backend` / `--linear-backend`	Kernel backend for MoE / dense layers (`auto` = let the oracle pick)	Forcing the wrong one: crashes or silent garbage; `auto` on a cubin-less wheel: `no kernel image`
`--kv-cache-dtype fp8`	Halves KV cost vs fp16	Model-specific; some models loop or degrade: validate per checkpoint
`--speculative-config`	Enables spec decoding (method, drafter, k)	Unsoaked configs crash under load (#41190); n-gram silently changes scheduler behavior
`--enable-auto-tool-choice` + `--tool-call-parser`	Lets agent clients send `tool_choice: auto`; parser must match the model family	Missing: every agent request 400s. Wrong parser: calls stall silently in `content`
`--reasoning-parser`	Splits `<think>` into `reasoning_content`	Missing: chain-of-thought leaks into the visible reply
`--enforce-eager`	Disables CUDA graphs	Was a needed workaround (hybrid-Mamba, ≤v0.20); now just a ~37% decode tax: drop it
`--enable-prefix-caching`	KV reuse for repeated prompt prefixes	On by default in V1, but some archs/features disable it: check, don’t assume (see #42084)
`--enable-chunked-prefill`	Interleaves prefill chunks with decode	Community-reported ~9× MoE slowdown on some SSM+MoE combos: A/B per model
`--attention-backend`	Pins the attention kernel	Forcing one crashed hybrid models on some builds; current nightlies handle `flashinfer` fine: test per image
`--mamba-ssm-cache-dtype`	SSM state cache precision on hybrid-Mamba models	`float32` is the validated setting in our Nemotron recipe

Sources & receipts

vLLM team, “vLLM on NVIDIA DGX Spark”: https://vllm.ai/blog/2026-06-01-vllm-dgx-spark
NVIDIA DGX Spark playbooks: https://build.nvidia.com/spark/vllm/instructions · https://github.com/NVIDIA/dgx-spark-playbooks
vLLM PRs referenced: #38126, #39825 (v0.20.0); #35568, #40082 (v0.22.0); #40923 (v0.22.1rc0). Upstream spec-decode IMA issues: #41190, #42084.
Community playbook with the broadest recipe coverage, the most useful single Spark reference we’ve found, whose Marlin guidance this post extends with checkpoint-format context: https://vlaicu.io/posts/dgx-vllm/
Community reports illustrating the software-immaturity-vs-hardware-limitation distinction this post draws: a slow-tokens/s thread where the cause is corrected from hardware to an immature kernel stack (https://forums.developer.nvidia.com/t/slow-performance-for-redhatai-qwen3-coder-next-nvfp4/363129), and the “SM121 software support is severely lacking” thread (https://forums.developer.nvidia.com/t/dgx-spark-sm121-software-support-is-severely-lacking-official-roadmap-needed/357663)
AEON-7 DFlash GB10 measurements: https://github.com/AEON-7/vllm-dflash · Z-Lab DFlash: https://github.com/z-lab/dflash
Checkpoints: unsloth/Qwen3.6-35B-A3B-NVFP4 (deployed), nvidia/Qwen3.6-35B-A3B-NVFP4, nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4, nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4
Bench methodology: single-stream prefill/decode script measuring TTFT, prefill tok/s, decode tok/s from server token counts, with prefix-cache-defeating nonces and forced full-length generation (ignore_eos), adapted from the vlaicu.io playbook §10.