Model Catalog模型部署配方与调优报告

模型

GLM 5.3 (NVFP4)

Tuned by LLM AutoTune: TP4 on four B300 GPUs, single node, modelopt_fp4 weights, EAGLE speculative decoding, hierarchical KV cache. Normalized throughput (8 GPUs) within the SLO vs the baseline variant: +195.3% (50k + 1.5k), +30.6% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.4

模型

MiMo-V2.5

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, fp8 KV cache, EAGLE speculative decoding. +121% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · 测于 v1.0.0

模型

DeepSeek-V4.1-Flash

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +89% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · 测于 v1.0.0

模型

Laguna-S-2.1

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, fp8 KV cache. +26% on the tuning benchmark vs the baseline variant.

8k + 1k tokens · 测于 v1.0.0

模型

Hy3

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, NEXTN speculative decoding. Normalized throughput within the SLO vs the baseline variant: +64.2% (50k + 1.5k), +7.6% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.0

模型

DeepSeek-V4-Flash-0731

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, DSPARK speculative decoding. Normalized throughput within the SLO vs the baseline variant: +58.0% (50k + 1.5k), +23.4% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.0

模型

Kimi K3

Tuned by LLM AutoTune: TP8 on eight B300 GPUs, single node, fp8 KV cache, DSPARK speculative decoding, hierarchical KV cache. Normalized throughput within the SLO vs the baseline variant: +57.6% (50k + 1.5k), +21.4% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.5

模型

Mistral-Small-4-119B-2603

Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +52.5% (50k + 1.5k), +32.1% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.0

模型

DeepSeek-V4-Flash

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, EAGLE speculative decoding. +51% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · 测于 v1.0.2

模型

gemma-4-26B-A4B-it

Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +38.8% (50k + 1.5k), +45.6% (8k + 1k).

8k + 1k tokens · 测于 v1.0.0

模型

gpt-oss-120b

Tuned by LLM AutoTune: TP2 DP4 on eight H100 GPUs, single node, fp8 KV cache, EAGLE3 speculative decoding (lmsys/EAGLE3-gpt-oss-120b-bf16). +44% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · 测于 v1.0.0

模型

google/gemma-4-31B-it

Tuned by LLM AutoTune: TP4 on four H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +32.4% (50k + 1.5k), +1.6% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.0

模型

Qwen3.8-27B-FP8

Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +30.1% (50k + 1.5k), +12.1% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.0

模型

Nemotron-3.5-Lightning-30B-A3B

Tuned by LLM AutoTune: TP1 DP8 on eight H100 GPUs, single node. +27% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · 测于 v1.0.0

模型

Nex-N2.5-Pro

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, fp8 KV cache. +24% on the tuning benchmark vs the baseline variant.

8k + 1k tokens · 测于 v1.0.0

模型

Ling-3.0-flash-VL

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +15% on the tuning benchmark vs the baseline variant.

8k + 1k tokens · 测于 v1.0.0

模型

Ling-3.0-flash

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +15.5% (50k + 1.5k), +21.3% (8k + 1k).

8k + 1k tokens · 测于 v1.0.0

模型

GLM-5.3-Flash

Tuned by LLM AutoTune: TP8 on eight B300 GPUs, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +12.5% (50k + 1.5k), +8.2% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.1

模型

Qwen3.6 35B-A3B

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +4.3% (50k + 1.5k), +10.0% (8k + 1k).

8k + 1k tokens · 测于 v1.1.0

模型

Qwen3.8-Flash-Next

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +6% on the tuning benchmark vs the baseline variant.

8k + 1k tokens · 测于 v1.0.0

模型

gpt-oss-20b

Tuned by LLM AutoTune: TP1 on one H100 GPU, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +6.2% (50k + 1.5k), +2.2% (8k + 1k).

50k + 1.5k tokens · 测于 v1.0.0

模型

GLM-5.3-Flash

Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +5% on the tuning benchmark vs the baseline variant.

50k + 1.5k tokens · 测于 v1.0.1

模型

Mistral-Nemo-Instruct

Tuned by LLM AutoTune: TP1 on one H100 GPU, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +1.4% (50k + 1.5k), +5.3% (8k + 1k).

8k + 1k tokens · 测于 v1.0.0

模型

DeepSeek-V4-Flash

TP4 on four GPUs. 149 GiB of weights do not need eight cards: halving them still leaves 61% of the KV pool, because the weights are a fixed cost that does not shrink with the card count. Measured 38.57 GiB of weights per card and max_total_num_tokens 4,002,048, against 6,524,672 at TP8. Those two numbers imply ~34 and ~68 KiB per token respectively, so neither can be used to extrapolate another TP width -- the reason for the gap is not known.

模型

DeepSeek-V4-Flash

TP8 on one eight-GPU node, default MoE runner. Measured max_total_num_tokens 6,524,672.

模型

DeepSeek-V4-Flash

Baseline: TP8 on eight H100 GPUs, single node.

模型

DeepSeek-V4-Flash

TP4 on four GPUs, the vLLM counterpart of sglang-tp4-6000d. Measured 39.42 GiB per card for weights plus non-torch memory, 33.07 GiB left for KV, GPU KV cache size 3,447,225 tokens -- 3.29x the 1,048,576-token maximum request length. The engine's own suggestion is to raise the KV allocation to 40.08 GiB with --kv-cache-memory, which is left off here because it has not been load-tested.

模型

DeepSeek-V4-Flash

TP8 on one eight-GPU RTX 6000D node. Single-node, so no RDMA and no host networking.

模型

DeepSeek-V4-Flash-0731

Baseline: TP8 on eight H100 GPUs, single node.

模型

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build with TP4 on four RTX 6000D GPUs. Measured: ready in ~6.5 min, 40.94 GB of weights per card, KV ~3.6M tokens. Two fit one node. --moe-runner-backend=marlin is required: the default trtllm backend has no sm120 kernels, and flashinfer_cutlass serves but silently miscomputes (identical temperature-0 requests answer 23*47 as 161, 1721, 1551); marlin was correct and stable. /root/.cache is a hostPath so compiled kernels survive a restart.

模型

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build with TP8 on one RTX 6000D node. Measured: ready in ~7 min on a cold cache, KV 6,274,304 tokens, deployed with this chart. --moe-runner-backend=marlin is required: the default trtllm backend has no sm120 kernels, and flashinfer_cutlass serves but silently miscomputes (identical temperature-0 requests answer 23*47 as 161, 1721, 1551); marlin was correct and stable. /root/.cache is a hostPath so compiled kernels survive a restart.

模型

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build (NVFP4 routed experts, FP8 elsewhere) with TP4 on four RTX 6000D GPUs, single node. FlashInfer autotune is off: with it on, vLLM v0.30.0 hangs on these cards at the whole-model autotune pass. /root/.cache is a hostPath so compiled kernels survive a restart.

模型

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build with TP8 on one RTX 6000D node. Measured in docker: ready in ~10 min on a cold cache, KV 5,222,157 tokens, answers correct; thinking on by default. FlashInfer autotune is off: with it on, vLLM v0.30.0 hangs on these cards at the whole-model autotune pass. /root/.cache is a hostPath so compiled kernels survive a restart.

模型

DeepSeek-V4-Flash-0731 (NVFP4)

NVIDIA's NVFP4 build, prefill/decode disaggregated: TP8 per instance on RTX 6000D nodes, NIXL transfer backend, 2 prefill + 1 decode, with L2 KV-offload to mooncakeStore (global segment 100GB, local buffer 4GB). Production-shape values pinned to the harbor v0.30.0-pd-fix2 image, 0.85 GPU memory utilization and 850Gi shm for the large KV offload; node affinity pins pods to RTX-6000D. ModelRoute deepseek-v4-flash publishes the service via the llm-route openresty gateway.

模型

DeepSeek-V4-Pro

TP8 per node, PP2 across two nodes over RoCE. The fp8 e4m3 weights need no MoE backend override. Measured max_total_num_tokens 2,833,152.

模型

DeepSeek-V4-Pro

TP8 per node, PP2 across two nodes over RoCE.

模型

DeepSeek-V4.1-Flash

Baseline: TP8 on eight H100 GPUs, single node.

模型

gemma-4-26B-A4B-it

Baseline: TP2 on two H100 GPUs, single node.

模型

GLM-5.3-Flash

Baseline: TP8 on eight B300 GPUs, single node.

模型

GLM-5.3-Flash

Baseline: TP8 on eight H100 GPUs, single node.

模型

GLM-5.2

TP8 per node, PP2 across two nodes over RoCE.

模型

GLM 5.3 (NVFP4)

Baseline: TP8 on eight B300 GPUs, single node, modelopt_fp4 weights.

模型

GLM 5.3 (NVFP4)

TP8 per node with expert parallel, PP2 across two nodes over RoCE. EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. KV pool 1,180,288 tokens, max_req_input_len 1,048,570, and a real 1M-token request completes in 428 s. At 50k/1.5k and concurrency 10: 4,341 input and 126 output tok/s; at 80k/600: 6,722 and 47.6, ahead of the vLLM variant while running 10 concurrent requests against its 17. TTFT is 1.7-3x worse throughout, because prefill queues in 8,192-token chunks.

模型

GLM 5.3 (NVFP4)

Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 EP8 per role on RTX 6000D over RoCE. EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).

模型

GLM 5.3 (NVFP4)

Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 EP8 per role on RTX 6000D over RoCE, with local KV offload via hicache (size 100). EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).

模型

GLM 5.3 (NVFP4)

TP8 per node, PP2 across two nodes over RoCE, on the same NVFP4 weights as the B300 variant.

模型

GLM 5.3 (NVFP4)

Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 per role on RTX 6000D, mooncake transfer backend. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).

模型

GLM 5.3 (NVFP4)

Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 per role on RTX 6000D, mooncake transfer backend + local KV offload via mooncakeStore (8 GPUs x 100 GB global segment + 8 x 4 GB local buffer per engine). Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).

模型

google/gemma-4-31B-it

Baseline: TP4 on four H100 GPUs, single node.

模型

gpt-oss-120b

Baseline: TP2 on two H100 GPUs, single node.

模型

gpt-oss-20b

Baseline: TP1 on one H100 GPU, single node.

模型

Hy3

Baseline: TP8 on eight H100 GPUs, single node.

模型

Kimi K2.5

TP8 per node, PP2 across two nodes over IB.

模型

Kimi K3

Baseline: TP8 on eight B300 GPUs, single node.

模型

Kimi K3

TP8 per node, PP4 across four nodes over RoCE. Marlin is the only correct MoE runner here: the default upcasts the MXFP4 experts to bf16 and needs ~6000 GiB, triton_kernel fails in the weight loader, and flashinfer_mxfp4 loads, serves and computes wrong answers. Measured 47.81 GiB of weights per card and max_total_num_tokens 1,237,195.

模型

Kimi K3

Kimi K3 with SGLang and vLLM on a single eight-GPU B300 node, or vLLM with TP8 and PP4 across four RTX 6000D nodes. The tuned SGLang B300 variant uses DCP8, an fp8 KV cache, DSPARK speculative decoding, and hierarchical KV caching. DSPARK requires engine-compatible Kimi-K3-DSpark draft weights on every eligible node, mounted beside the target model.

模型

Kimi K3

TP8 per node, PP4 across four nodes over RoCE. A Marlin MoE path is required on sm120, to keep the 4-bit weights 4-bit; the default runner upcasts them to bf16 and does not fit. sglang reaches the same shape with --moe-runner-backend marlin.

模型

Laguna-S-2.1

Baseline: TP4 on four H100 GPUs, single node.

模型

Ling-3.0-flash

Baseline: TP8 on eight H100 GPUs, single node.

模型

Ling-3.0-flash-VL

Baseline: TP8 on eight H100 GPUs, single node.

模型

MiMo-V2.5

Baseline: TP8 on eight H100 GPUs, single node.

模型

Mistral-Nemo-Instruct

Baseline: TP1 on one H100 GPU, single node.

模型

Mistral-Small-4-119B-2603

Baseline: TP2 on two H100 GPUs, single node.

模型

Mock LLM

mock-llmd on CPU: no GPU, no weights. For tests only -- the responses are synthetic.

模型

Nemotron-3.5-Lightning-30B-A3B

Baseline: TP1 on one H100 GPU, single node.

模型

Nex-N2.5-Pro

Baseline: TP8 on eight H100 GPUs, single node.

模型

Qwen3-8B

Baseline: one Ascend 910B card, single node. vLLM's Ascend backend (CANN), not a CUDA build. Not yet tuned -- no AutoTune run on this hardware.

模型

Qwen3-8B

Baseline: bf16, TP1 on one NVIDIA A100 SXM4 80GB GPU, single node. Not yet tuned on this hardware.

模型

Qwen3.6 35B-A3B

TP2 on two H100 or H800 GPUs, single node.

模型

Qwen3.6 35B-A3B

Baseline: TP8 on eight H100 GPUs, single node.

模型

Qwen3.8-27B-FP8

Baseline: TP1 on one H100 GPU, single node.

模型

Qwen3.8-Flash-Next

Baseline: TP8 on eight H100 GPUs, single node.