Take LLM inference from a working prototype to reliable production, and turn every GPU into an efficient token factory.
Built on Kubernetes, ModelSphere brings model deployment, cache-aware routing, SLO-based autoscaling, comprehensive observability, recovery from hangs within seconds and the AutoTune feedback loop together in a production-grade inference scheduling system that reliably handles large-scale traffic on heterogeneous hardware.
Standard APIs for a wide range of agent frameworks.
Production routing, SLO autoscaling, seamless stream recovery, and engine orchestration.
Manage compute and topology across chip ecosystems without vendor lock-in.
Architectural Pillars
Built for production engineering. ModelSphere combines advances in research with experience operating large-scale services in a standardized platform.
Set up an AI inference platform for production in minutes, with model serving, scheduling, autoscaling, observability, and operations built in. Manage your cluster through a unified Web UI without complex low-level tuning.
ModelSphere runs across heterogeneous AI hardware through one platform. It natively supports general-purpose GPUs and uses EngineX to integrate Huawei Ascend, Iluvatar CoreX, Moore Threads, and Hygon accelerators, bringing mixed-hardware nodes under unified management.
Keep pace with the model ecosystem through our inference optimization community, adapting and tuning new models for rapid import, configuration, and production deployment.
ModelSphere continuously integrates new inference optimizations. Built-in AutoTune adapts configurations to your models, hardware, and production workloads to improve latency, throughput, and resource efficiency.
The client maintains a persistent SSE connection; the gateway routes the request to Model service A, which emits tokens 1 → 2 → 3 → 4 → 5.
Model service A encounters a GPU memory deadlock or CUDA illegal memory access error, interrupting generation. Hang-Watcher detects a token emission timeout within 140 ms and tells the gateway to close its connection to A.
The gateway captures the emitted tokens 1–5 and uses a shared handle to the L3 KV cache to immediately redirect the request to a healthy standby node, Model service B。
Model service B quickly resumes generation at tokens 6 → 7 → 8... The gateway appends the output to the original SSE stream, preserving all text without a visible interruption.
Move beyond static TPM/RPM limits with real-time production feedback based on time to first token (TTFT) and output tokens per second (OTPS).
Static rate limits cannot detect the cascading queues caused by saturated prefill capacity. ModelSphere monitors time to first token (TTFT). When it exceeds the threshold, the gateway responds within milliseconds with HTTP 429 and Retry-After: 120ms to prompt client backoff and prevent cluster overload.
Adjust inference Pod replicas within milliseconds as concurrency changes. During off-peak hours, high-priority models use idle accelerators, raising overall GPU utilization from the industry average of 45% to over 85% in this design example. In-flight requests finish before replicas are released during scale-down.
ModelSphere connects real production traffic, idle off-peak compute, hyperparameter grid search, and zero-downtime canary releases in an automated optimization loop, reducing manual trial-and-error benchmarking.
Use idle compute to unlock hardware potential and make inference faster over time.
Static test datasets → Real traffic features from a 7-day sliding window
Benchmarks consuming production resources → Concurrent grid search on idle off-peak compute
Restarting services to update parameters → Automatically validate the best configuration and hot-load it in production without downtime
Reliable, low-latency, highly available service
Deploy the best configuration with seamless hot reload
Build sliding-window datasets and prefix topology
Select the best inference configuration using real workloads
Search inference hyperparameters in parallel during off-peak hours
Publish benchmark input and output distributions, hardware topology, and trade-offs so every tuning gain can be verified.
| Model and architecture | Hardware | Workload (Prompt + Output) | Throughput (Default vs Tuned) | Performance gain | Tuning techniques and trade-offs |
|---|---|---|---|---|---|
| MiMo-V2.5 | 8 × NVIDIA H100 | 50k + 1.5k (long context) | 13,900 → 30,600 tok/s | +119.9% | Concurrency 4 → 26; TTFT 4.81s → 7.66s (within the SLO of ≤ 8s); GPQA accuracy decreases by 3.03% (an explicit trade-off) |
| DeepSeek-V4.1-Flash | 8 × NVIDIA H100 | 50k + 1.5k / 8k + 1k | Large long-context gains / steady on common workloads | +89.5% / +5.0% | Gains depend on workload; AutoTune adapts chunk sizes for very long contexts |
| Qwen3.6-35B-A3B | 2 × NVIDIA H800 | Agentic (c=8, concurrent multi-turn) | 6,200 → 8,016 tok/s | +29.3% | SGLang + FlashAttention-3 + FP8 + Chunked Prefill 32768 + extra_buffer |
| GLM-5.3 (NVFP4) | 4 × NVIDIA B300 | Mixed workload simulating production traffic | Baseline comparison → Peak gain | +195.3% | AutoTune selects the optimal single-node TP4 topology and integrates modelopt_fp4 kernels |
Explore the open source code and build the next generation of inference infrastructure with AI engineers.
