Model CatalogModel deployment recipes and tuning reports

Model

GLM 5.3 (NVFP4)

Tuned by LLM AutoTune: TP4 on four B300 GPUs, single node, modelopt_fp4 weights, EAGLE speculative decoding, hierarchical KV cache. Normalized throughput (8 GPUs) within the SLO vs the baseline variant: +195.3% (50k + 1.5k), +30.6% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.4

Model

MiMo-V2.5

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, fp8 KV cache, EAGLE speculative decoding. +121% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · Measured with v1.0.0

Model

DeepSeek-V4.1-Flash

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +89% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · Measured with v1.0.0

Model

Laguna-S-2.1

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, fp8 KV cache. +26% on the tuning benchmark vs the baseline variant.

8k + 1k tokens · Measured with v1.0.0

Model

Hy3

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, NEXTN speculative decoding. Normalized throughput within the SLO vs the baseline variant: +64.2% (50k + 1.5k), +7.6% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.0

Model

DeepSeek-V4-Flash-0731

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, DSPARK speculative decoding. Normalized throughput within the SLO vs the baseline variant: +58.0% (50k + 1.5k), +23.4% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.0

Model

Kimi K3

Tuned by LLM AutoTune: TP8 on eight B300 GPUs, single node, fp8 KV cache, DSPARK speculative decoding, hierarchical KV cache. Normalized throughput within the SLO vs the baseline variant: +57.6% (50k + 1.5k), +21.4% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.5

Model

Mistral-Small-4-119B-2603

Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +52.5% (50k + 1.5k), +32.1% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.0

Model

DeepSeek-V4-Flash

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, EAGLE speculative decoding. +51% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · Measured with v1.0.2

Model

gemma-4-26B-A4B-it

Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +38.8% (50k + 1.5k), +45.6% (8k + 1k).

8k + 1k tokens · Measured with v1.0.0

Model

gpt-oss-120b

Tuned by LLM AutoTune: TP2 DP4 on eight H100 GPUs, single node, fp8 KV cache, EAGLE3 speculative decoding (lmsys/EAGLE3-gpt-oss-120b-bf16). +44% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · Measured with v1.0.0

Model

google/gemma-4-31B-it

Tuned by LLM AutoTune: TP4 on four H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +32.4% (50k + 1.5k), +1.6% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.0

Model

Qwen3.8-27B-FP8

Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +30.1% (50k + 1.5k), +12.1% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.0

Model

Nemotron-3.5-Lightning-30B-A3B

Tuned by LLM AutoTune: TP1 DP8 on eight H100 GPUs, single node. +27% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · Measured with v1.0.0

Model

Nex-N2.5-Pro

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, fp8 KV cache. +24% on the tuning benchmark vs the baseline variant.

8k + 1k tokens · Measured with v1.0.0

Model

Ling-3.0-flash-VL

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +15% on the tuning benchmark vs the baseline variant.

8k + 1k tokens · Measured with v1.0.0

Model

Ling-3.0-flash

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +15.5% (50k + 1.5k), +21.3% (8k + 1k).

8k + 1k tokens · Measured with v1.0.0

Model

GLM-5.3-Flash

Tuned by LLM AutoTune: TP8 on eight B300 GPUs, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +12.5% (50k + 1.5k), +8.2% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.1

Model

Qwen3.6 35B-A3B

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +4.3% (50k + 1.5k), +10.0% (8k + 1k).

8k + 1k tokens · Measured with v1.1.0

Model

Qwen3.8-Flash-Next

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +6% on the tuning benchmark vs the baseline variant.

8k + 1k tokens · Measured with v1.0.0

Model

gpt-oss-20b

Tuned by LLM AutoTune: TP1 on one H100 GPU, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +6.2% (50k + 1.5k), +2.2% (8k + 1k).

50k + 1.5k tokens · Measured with v1.0.0

Model

GLM-5.3-Flash

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +5% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · Measured with v1.0.1

Model

Mistral-Nemo-Instruct

Tuned by LLM AutoTune: TP1 on one H100 GPU, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +1.4% (50k + 1.5k), +5.3% (8k + 1k).

8k + 1k tokens · Measured with v1.0.0

Model

DeepSeek-V4-Flash

TP4 on four GPUs. 149 GiB of weights do not need eight cards: halving them still leaves 61% of the KV pool, because the weights are a fixed cost that does not shrink with the card count. Measured 38.57 GiB of weights per card and max_total_num_tokens 4,002,048, against 6,524,672 at TP8. Those two numbers imply ~34 and ~68 KiB per token respectively, so neither can be used to extrapolate another TP width -- the reason for the gap is not known.

Model

DeepSeek-V4-Flash

TP8 on one eight-GPU node, default MoE runner. Measured max_total_num_tokens 6,524,672.

Model

DeepSeek-V4-Flash

Baseline: TP8 on eight H100 GPUs, single node.

Model

DeepSeek-V4-Flash

TP4 on four GPUs, the vLLM counterpart of sglang-tp4-6000d. Measured 39.42 GiB per card for weights plus non-torch memory, 33.07 GiB left for KV, GPU KV cache size 3,447,225 tokens -- 3.29x the 1,048,576-token maximum request length. The engine's own suggestion is to raise the KV allocation to 40.08 GiB with --kv-cache-memory, which is left off here because it has not been load-tested.

Model

DeepSeek-V4-Flash

TP8 on one eight-GPU RTX 6000D node. Single-node, so no RDMA and no host networking.

Model

DeepSeek-V4-Flash-0731

Baseline: TP8 on eight H100 GPUs, single node.

Model

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build with TP4 on four RTX 6000D GPUs. Measured: ready in ~6.5 min, 40.94 GB of weights per card, KV ~3.6M tokens. Two fit one node. --moe-runner-backend=marlin is required: the default trtllm backend has no sm120 kernels, and flashinfer_cutlass serves but silently miscomputes (identical temperature-0 requests answer 23*47 as 161, 1721, 1551); marlin was correct and stable. /root/.cache is a hostPath so compiled kernels survive a restart.

Model

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build with TP8 on one RTX 6000D node. Measured: ready in ~7 min on a cold cache, KV 6,274,304 tokens, deployed with this chart. --moe-runner-backend=marlin is required: the default trtllm backend has no sm120 kernels, and flashinfer_cutlass serves but silently miscomputes (identical temperature-0 requests answer 23*47 as 161, 1721, 1551); marlin was correct and stable. /root/.cache is a hostPath so compiled kernels survive a restart.

Model

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build (NVFP4 routed experts, FP8 elsewhere) with TP4 on four RTX 6000D GPUs, single node. FlashInfer autotune is off: with it on, vLLM v0.30.0 hangs on these cards at the whole-model autotune pass. /root/.cache is a hostPath so compiled kernels survive a restart.

Model

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build with TP8 on one RTX 6000D node. Measured in docker: ready in ~10 min on a cold cache, KV 5,222,157 tokens, answers correct; thinking on by default. FlashInfer autotune is off: with it on, vLLM v0.30.0 hangs on these cards at the whole-model autotune pass. /root/.cache is a hostPath so compiled kernels survive a restart.

Model

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build, prefill/decode disaggregated: TP8 per instance on RTX 6000D nodes, NIXL transfer backend, 2 prefill + 1 decode, with L2 KV-offload to mooncakeStore (global segment 100GB, local buffer 4GB). Production-shape values pinned to the harbor v0.30.0-pd-fix2 image, 0.85 GPU memory utilization and 850Gi shm for the large KV offload; node affinity pins pods to RTX-6000D. ModelRoute deepseek-v4-flash publishes the service via the llm-route openresty gateway.

Model

DeepSeek-V4-Pro

TP8 per node, PP2 across two nodes over RoCE. The fp8 e4m3 weights need no MoE backend override. Measured max_total_num_tokens 2,833,152.

Model

DeepSeek-V4-Pro

TP8 per node, PP2 across two nodes over RoCE.

Model

DeepSeek-V4.1-Flash

Baseline: TP8 on eight H100 GPUs, single node.

Model

gemma-4-26B-A4B-it

Baseline: TP2 on two H100 GPUs, single node.

Model

GLM-5.3-Flash

Baseline: TP8 on eight B300 GPUs, single node.

Model

GLM-5.3-Flash

Baseline: TP8 on eight H100 GPUs, single node.

Model

GLM-5.2

TP8 per node, PP2 across two nodes over RoCE.

Model

GLM 5.3 (NVFP4)

Baseline: TP8 on eight B300 GPUs, single node, modelopt_fp4 weights.

Model

GLM 5.3 (NVFP4)

TP8 per node with expert parallel, PP2 across two nodes over RoCE. EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. KV pool 1,180,288 tokens, max_req_input_len 1,048,570, and a real 1M-token request completes in 428 s. At 50k/1.5k and concurrency 10: 4,341 input and 126 output tok/s; at 80k/600: 6,722 and 47.6, ahead of the vLLM variant while running 10 concurrent requests against its 17. TTFT is 1.7-3x worse throughout, because prefill queues in 8,192-token chunks.

Model

GLM 5.3 (NVFP4)

Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 EP8 per role on RTX 6000D over RoCE. EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).

Model

GLM 5.3 (NVFP4)

Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 EP8 per role on RTX 6000D over RoCE, with local KV offload via hicache (size 100). EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).

Model

GLM 5.3 (NVFP4)

TP8 per node, PP2 across two nodes over RoCE, on the same NVFP4 weights as the B300 variant.

Model

GLM 5.3 (NVFP4)

Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 per role on RTX 6000D, mooncake transfer backend. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).

Model

GLM 5.3 (NVFP4)

Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 per role on RTX 6000D, mooncake transfer backend + local KV offload via mooncakeStore (8 GPUs x 100 GB global segment + 8 x 4 GB local buffer per engine). Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).

Model

google/gemma-4-31B-it

Baseline: TP4 on four H100 GPUs, single node.

Model

gpt-oss-120b

Baseline: TP2 on two H100 GPUs, single node.

Model

gpt-oss-20b

Baseline: TP1 on one H100 GPU, single node.

Model

Hy3

Baseline: TP8 on eight H100 GPUs, single node.

Model

Kimi K2.5

TP8 per node, PP2 across two nodes over IB.

Model

Kimi K3

Baseline: TP8 on eight B300 GPUs, single node.

Model

Kimi K3

TP8 per node, PP4 across four nodes over RoCE. Marlin is the only correct MoE runner here: the default upcasts the MXFP4 experts to bf16 and needs ~6000 GiB, triton_kernel fails in the weight loader, and flashinfer_mxfp4 loads, serves and computes wrong answers. Measured 47.81 GiB of weights per card and max_total_num_tokens 1,237,195.

Model

Kimi K3

Kimi K3 with SGLang and vLLM on a single eight-GPU B300 node, or vLLM with TP8 and PP4 across four RTX 6000D nodes. The tuned SGLang B300 variant uses DCP8, an fp8 KV cache, DSPARK speculative decoding, and hierarchical KV caching. DSPARK requires engine-compatible Kimi-K3-DSpark draft weights on every eligible node, mounted beside the target model.

Model

Kimi K3

TP8 per node, PP4 across four nodes over RoCE. A Marlin MoE path is required on sm120, to keep the 4-bit weights 4-bit; the default runner upcasts them to bf16 and does not fit. sglang reaches the same shape with --moe-runner-backend marlin.

Model

Laguna-S-2.1

Baseline: TP4 on four H100 GPUs, single node.

Model

Ling-3.0-flash

Baseline: TP8 on eight H100 GPUs, single node.

Model

Ling-3.0-flash-VL

Baseline: TP8 on eight H100 GPUs, single node.

Model

MiMo-V2.5

Baseline: TP8 on eight H100 GPUs, single node.

Model

Mistral-Nemo-Instruct

Baseline: TP1 on one H100 GPU, single node.

Model

Mistral-Small-4-119B-2603

Baseline: TP2 on two H100 GPUs, single node.

Model

Mock LLM

mock-llmd on CPU: no GPU, no weights. For tests only -- the responses are synthetic.

Model

Nemotron-3.5-Lightning-30B-A3B

Baseline: TP1 on one H100 GPU, single node.

Model

Nex-N2.5-Pro

Baseline: TP8 on eight H100 GPUs, single node.

Model

Qwen3-8B

Baseline: one Ascend 910B card, single node. vLLM's Ascend backend (CANN), not a CUDA build. Not yet tuned -- no AutoTune run on this hardware.

Model

Qwen3-8B

Baseline: bf16, TP1 on one NVIDIA A100 SXM4 80GB GPU, single node. Not yet tuned on this hardware.

Model

Qwen3.6 35B-A3B

TP2 on two H100 or H800 GPUs, single node.

Model

Qwen3.6 35B-A3B

Baseline: TP8 on eight H100 GPUs, single node.

Model

Qwen3.8-27B-FP8

Baseline: TP1 on one H100 GPU, single node.

Model

Qwen3.8-Flash-Next

Baseline: TP8 on eight H100 GPUs, single node.