ModelSphere Helm Charts
Helm charts for running large language models on Kubernetes.
helm repo add modelsphere https://modelsphere.github.io/helm-charts
helm repo update
helm search repo modelsphere
| Chart | Version | App version | Description |
|---|---|---|---|
modelsphere/autoconfig | 0.4.3 | 0.4.3 | k8s 后端发现 → 自动同步 openresty peers / cache-aware-router workers(ModelRoute controller) |
modelsphere/bodylog | 0.1.16 | 0.1.15 | Body-log listener for the OpenResty router: receives full request/response frames over TCP, writes them as hourly JSONL, and serves rollups over HTTP |
modelsphere/bodylog-exporter | 0.3.7 | 0.2.14 | Turns the body-log listener's detail records into Prometheus metrics, and polls the router's control-plane endpoints for live routing state |
modelsphere/bodylog-victorialogs | 0.1.1 | v1.53.0 | Searchable body logs: Vector ships bodylog's JSONL into VictoriaLogs, registered as a Grafana datasource |
modelsphere/cart | 0.2.4 | 0.6.6 | Cache-aware router: routes each request to the replica that already holds the longest matching prompt prefix, while keeping load balanced |
modelsphere/catalog-mirror | 0.2.0 | 0.2.0 | In-cluster copy of the model catalog and this helm chart repository, for clusters whose route to GitHub is slow or unreliable; swiss is pointed at it instead of GitHub Pages |
modelsphere/console | 0.2.2 | 0.2.2 | ModelSphere community portal — identity (users, roles, login) and a federation gateway to Swiss and other backends. |
modelsphere/continuation-gateway | 0.1.4 | 0.2.7 | Continuation gateway: when a streaming completion stalls or drops mid-generation, resumes it once from what the client already received, so the client sees one continuous, complete stream |
modelsphere/llm-autotune | 0.3.1 | 0.3.0 | Automated tuning of LLM inference deployments — campaigns, a launch stack, and policy-as-code search |
modelsphere/llm-bench | 0.3.0 | 0.3.0 | LLM Benchmark Platform - FastAPI backend, Vue frontend, Dramatiq workers |
modelsphere/llm-slo-decision-gen | 0.3.9 | 0.8.1 | LLM and async-job SLO stack: decision-gen (recommendation service), optional slo-api (SLO storage HTTP/UI), and the LLMSLORequirement and JobSLORequirement CRDs under inference.modelsphere.dev. CRD ownership: this chart owns llmslorequirements.inference.modelsphere.dev and jobslorequirements.inference.modelsphere.dev (see crds/). Both CRDs carry helm.sh/resource-policy: keep so uninstall leaves them in the cluster — existing LLMSLORequirement / JobSLORequirement objects are not deleted with the release. Do not install a second chart that also ships these CRDs into the same cluster. LLMScaler CRD and the operator that reconciles it are provided by a separate chart (llmscaleoperator / llmscaleoperator-system). Deploy that one first if you need autoscaling, not just SLO storage. |
modelsphere/llmscaleoperator | 0.5.0 | 0.5.0 | Kubernetes operator that autoscales LLM inference workloads on LLM-specific signals -- KV-cache utilization, queue depth, TPM/capacity load -- rather than CPU and memory. An HPA specialized for token serving. Ships the LLMScaler CRD (autoscaling.modelsphere.dev/v1alpha1), the LLMService CRD (serving.modelsphere.dev/v1alpha1) and the controller that reconciles them. The SLO storage side lives in the llm-slo-decision-gen chart; the two are independent, and this one is what you need for autoscaling alone. Generated from the llm-operator repo (dist/chart, kubebuilder helm plugin) and maintained here, because this is where charts are published from. |
modelsphere/minimax-h3 | 0.2.2 | 0.2.2 | Self-hosted MiniMax H3-compatible video generation for the open-weight MiniMax H3 models -- H3 Router, H3 Server and SGLang in one release, with separate FL2VA and Ref2VA pools |
modelsphere/mooncake-master | 0.1.0 | 0.3.13 | Highly available Mooncake Store masters: two or more mooncake_master replicas electing a leader through etcd (bundled or external), with an optional sidecar that re-registers NVMe-oF volumes on whichever master wins |
modelsphere/openresty | 0.1.28 | 0.1.28 | Session-affinity router for LLM inference backends: pins a conversation to the backend that already holds its prefix cache, with active health checking, adaptive concurrency and full body logging |
modelsphere/rdma-injector | 0.2.3 | 0.2.0 | Mutating admission webhook that spares workload manifests the RDMA boilerplate: pods labelled rdma-ib:"true" get the /etc/gpu-node mount and an NCCL_IB_HCA sourced from the node's healthy InfiniBand ports injected automatically. The webhook code is baked into the image; its serving certificate is self-signed by the chart at install time, so it installs into any namespace. |
modelsphere/sglang | 0.9.37 | 1.0.0 | SGLang inference deployment -- single-node, multi-node (LeaderWorkerSet) or prefill/decode-disaggregated (DisaggregatedSet or merged colocate pod), with optional cache-aware routing, autoscaling and hang detection |
modelsphere/swiss | 0.7.2 | 0.7.2 | swissd -- the Swiss deploy control plane for the sglang and vllm charts. One instance per cluster, running inside the cluster it manages: it reads the model catalog over HTTP, reads this cluster's releases and nodes through its own ServiceAccount, and serves the web UI embedded in its binary. |
modelsphere/vllm | 0.9.8 | 1.0.0 | vLLM inference deployment -- single-node or multi-node (LeaderWorkerSet), with optional cache-aware routing, autoscaling and hang detection |