Vulavula Logo
Integrations

Self-host (K8s/Openshift)

Enterprise only!

Self-deployment of the Vulavula inference stack is intended for enterprise customers running the platform inside their own infrastructure. It is not the right path for most users — the managed Vulavula API is faster to integrate, fully maintained, and requires no cluster or GPU operations on your side.

Please reach out first!

Before you attempt a self-deploy, contact us at [email protected] so we can assist with sizing, licensing, access to the private GHCR registry, and ongoing support. Deploying without that conversation is unsupported and will run into private-image auth, chart-version pinning, and feature-gate prerequisites you'll otherwise discover the hard way.

What this is

The your-cloud Helm chart deploys the full Lelapa inference stack — the same one powering the managed Vulavula API — into your own Kubernetes cluster. You bring the cluster and GPU capacity; the chart ships everything else needed to run speech-to-text and translation workloads.

It front-loads a small FastAPI gateway (inference-api) that accepts your requests and routes them, via a chart-owned internal Envoy proxy (inference-router), to two GPU-backed Triton predictor pools. The two pools cover the two distinct workloads the API supports: low-latency streaming (live) and higher-throughput batch processing (sync). Models travel as OCI images pulled with the same credentials as the app images — no separate S3 or model-registry plumbing on your side.

This chart does not expose anything outside the cluster or attach authentication — bring your own Ingress / Gateway / LoadBalancer and auth layer in front. See the chart's own README.md, RUNBOOK.md, and DESIGN.md (shipped inside the chart tarball at charts/your-cloud/) for the full, maintained reference. The steps below are a quickstart; reach out to us first (see the callout above) to get access.

Prerequisites

Before you start, confirm with us which your-cloud chart version and which their-cloud-mvp image release tag you should pin — both are private (GHCR internal visibility) and version advice is per-customer.

  • Kubernetes ImageVolume feature gate (alpha since 1.31, off by default) — the predictor pods mount model weights straight out of an OCI image; there's no fallback path. Confirm the gate is actually on with whoever manages the cluster.
  • At least one GPU in the cluster for inference (plan for 3 GPUs of headroom at default autoscaling limits — live scales up to 2 replicas × 1 GPU, sync is pinned to 1 replica × 1 GPU). Set inferenceServer.{live,sync}.nodePool.{selector,tolerations} in your values file to match your actual GPU node's labels/taints — the chart ships no default here, leaving it unset means predictor pods get no selector/toleration at all.
  • NVIDIA drivers + device plugin already on those GPU nodes, so nvidia.com/gpu shows up as allocatable. This is a real, easy-to-miss prerequisite on OpenShift specifically — neither ARO nor (per NVIDIA's own docs) ROSA ships GPU drivers by default, unlike some managed AKS/EKS node images. You'll separately need to install Node Feature Discovery + the NVIDIA GPU Operator from OperatorHub (a Subscription for each, plus a ClusterPolicy CR) before nvidia.com/gpu becomes allocatable at all:
    TerminalCode
    kubectl describe node <gpu-node-name> | grep -A5 Allocatable # look for nvidia.com/gpu in the output
    Driver installation can trigger a node reboot the first time it loads — expect a NodeNotReady/NetworkPluginNotReady blip on that node for a couple of minutes; it self-heals once the node comes back.
  • kubectl or oc, plus helm v3.x, pointed at the target cluster. Every command below is written for kubectl/plain Kubernetes; on OpenShift substitute oc 1:1 for kubectl (e.g. oc get pods instead of kubectl get pods) and additionally grant the restricted-v2 SCC to the service account your pods run as — see the OpenShift callout after Step 7.
  • A lelapa-ghcr image pull secret in the target namespace — the app, runtime, and model weights images are all private (GHCR internal). Create it with a GitHub PAT that has read:packages scope (Step 7 below); the chart never provisions it itself.
  • Your own Ingress / Gateway + auth in front of the chart's internal :9000 service — the chart does not expose anything outside the cluster, terminate TLS, or attach authentication. Its only built-in auth is a Basic Auth pair on the service itself, not a substitute for real edge auth.

cert-manager and KServe

cert-manager and KServe are also required, but they're installed as part of the steps below — skip those steps if your cluster already has them.

Install

  1. Get a chart version and image tag from us

    Email [email protected] for the your-cloud chart version (X.Y.Z) and the their-cloud-mvp image release tag (YY.MM.<counter>, e.g. 26.7.2) you should pin. Both artifacts live under private GHCR packages.

  2. Install cert-manager (skip if your cluster already has it)

    TerminalCode
    kubectl get ns cert-manager || { kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.18.2/cert-manager.yaml kubectl wait --for=condition=Available --timeout=180s -n cert-manager \ deployment/cert-manager deployment/cert-manager-cainjector deployment/cert-manager-webhook }
  3. Install KServe (CRDs + controller) (skip if already installed)

    Two separate helm install calls — the CRD chart must land and become discoverable before the controller chart runs:

    TerminalCode
    helm upgrade --install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd \ --version v0.18.0 --namespace kserve --create-namespace helm upgrade --install kserve-resources oci://ghcr.io/kserve/charts/kserve-resources \ --version v0.18.0 --namespace kserve \ --set kserve.service.serviceClusterIPNone=true kubectl wait --for=condition=Available --timeout=180s -n kserve deployment/kserve-controller-manager
  4. Log in to the Lelapa Helm OCI registry

    TerminalCode
    helm registry login ghcr.io -u <github-user> -p <github-token-with-read:packages>
  5. Configure your my-values.yaml

    Minimum every install needs to touch:

    YAMLCode
    namespace: <namespace> # where you're deploying image: tag: "<X.Y.Z>" # shared release tag for gateway/triton/models — pin an immutable release tag, not a rolling one inferenceServer: live: enabled: true nodePool: selector: {} # match your actual GPU node labels (Step 1 prereq) tolerations: [] sync: enabled: true nodePool: selector: {} tolerations: [] secrets: mode: directValues # existingSecret | directValues — directValues lets the chart render the Secret itself directValues: {} # leave CLIENT_USERNAME/CLIENT_PASSWORD unset to auto-generate random Basic Auth on first install persistence: meteringDb: enabled: true storageClass: ""

    To see the full set of available keys (or check what any of the above defaults to), render the default values straight from the published chart:

    TerminalCode
    helm show values oci://ghcr.io/lelapa-ai/charts/your-cloud --version <X.Y.Z>
  6. Install your-cloud

    Dry-run against the real cluster first — this catches CRD/webhook mismatches that helm template alone won't, since it actually talks to the API server:

    TerminalCode
    helm upgrade --install your-cloud oci://ghcr.io/lelapa-ai/charts/your-cloud \ --version <X.Y.Z> \ -f my-values.yaml \ --namespace <namespace> --create-namespace \ --dry-run=server

    Once that looks right, drop --dry-run=server:

    TerminalCode
    helm upgrade --install your-cloud oci://ghcr.io/lelapa-ai/charts/your-cloud \ --version <X.Y.Z> \ -f my-values.yaml \ --namespace <namespace> --create-namespace

    Match --namespace to whatever namespace: you set in my-values.yaml — Helm tracks the release itself in whatever --namespace you pass here, independently of .Values.namespace governing where the chart's resources land. A mismatch doesn't break the install, but it does mean helm list / helm uninstall won't find the release where you expect it.

    The chart's preflight check (on by default; set --set preflight.enabled=false to bypass) fails the install immediately with a clear message if the KServe CRD isn't registered or — on upgrade — if the predictor Services aren't actually headless.

    OpenShift: SCC

    Grant the restricted-v2 SCC to the service account your pods run as (default, unless you've set one explicitly) — without this, pods fail admission on OpenShift:

    TerminalCode
    oc adm policy add-scc-to-user restricted-v2 -z default -n <namespace>
  7. Create the lelapa-ghcr image pull secret

    TerminalCode
    kubectl create secret docker-registry lelapa-ghcr \ --docker-server=ghcr.io \ --docker-username=<github-user> \ --docker-password=<github-token-with-read:packages> \ --namespace <namespace>

    Needed for the gateway/triton/models images to actually pull — the chart never provisions this itself, and pods sit ImagePullBackOff without it, even though the install above succeeds.

    Also link the secret to the pipeline-predictors service account

    Predictor pods mount model weights via Kubernetes' native image volume type (not a regular container image pull) — on this cluster's Kubernetes/CRI-O version, that volume type does not pick up imagePullSecrets set at the pod spec level, only secrets linked to the pod's service account. Without this extra step, predictor pods sit Init:ImagePullBackOff on the models-image-tracker initContainer with unable to retrieve auth token: invalid username/password: unauthorized, even though the same secret already lets the gateway/triton container images pull fine:

    TerminalCode
    kubectl create secret docker-registry lelapa-ghcr ... # as above # OpenShift: oc secrets link pipeline-predictors lelapa-ghcr --for=pull -n <namespace> # plain Kubernetes: kubectl patch serviceaccount pipeline-predictors -n <namespace> \ -p '{"imagePullSecrets": [{"name": "lelapa-ghcr"}]}'
  8. Verify

    TerminalCode
    kubectl get isvc,deploy,pods -n <namespace> kubectl get pods -n <namespace> -w # watch until Running/Ready kubectl port-forward -n <namespace> svc/inference-api 9000:9000 curl localhost:9000/health # expect "OK"

    Check both predictor pools are ready (KServe sets .status.conditions):

    TerminalCode
    kubectl get isvc -n <namespace> -o wide kubectl describe isvc pipeline-live -n <namespace> kubectl describe isvc pipeline-sync -n <namespace>

    Check the internal gateway (a plain chart-owned Deployment, so an ordinary rollout check — no external controller/CRD status to wait on):

    TerminalCode
    kubectl get pods -n <namespace> -l app=inference-router

    helm install/upgrade prints this same checklist via templates/NOTES.txt — re-run helm get notes your-cloud -n <namespace> any time to see it again. Once everything above is Running/Ready, see Qualification below to confirm the deployment is actually serving correct, on-spec results — not just that the pods are up.

External exposure and auth are your responsibility

This chart does not create an Ingress / Gateway / LoadBalancer or attach authentication in front of inference-api (:9000, ClusterIP). Front it with your own, and terminate TLS there — the chart-rendered gateway-secrets Basic Auth pair travels in the clear otherwise.

Update

TerminalCode
helm diff upgrade your-cloud oci://ghcr.io/lelapa-ai/charts/your-cloud \ --version <X.Y.Z> \ -f my-values.yaml -n <namespace> # requires the helm-diff plugin helm upgrade --install your-cloud oci://ghcr.io/lelapa-ai/charts/your-cloud \ --version <X.Y.Z> \ -f my-values.yaml -n <namespace>

Same command for first install and every subsequent upgrade — helm upgrade --install is idempotent. Bump only the chart --version to move chart templates / schema / defaults; bump only image.tag in your values file to move what code / model weights run. Tell us when you intend to do either so we can confirm version compatibility.

Rollback

TerminalCode
helm history your-cloud -n <namespace> helm rollback your-cloud <REVISION> -n <namespace>

Rollback reverts the chart's rendered manifests to a previous release, not any data — the metering-db PVC (if enabled) isn't touched either way.

Uninstall

TerminalCode
helm uninstall your-cloud -n <namespace>

The metering-db-pvc PVC is not deleted automatically by Helm — remove it manually if you actually want the data gone:

TerminalCode
kubectl delete pvc metering-db-pvc -n <namespace>

Troubleshooting

  • Predictor pod stuck Pending, no obvious reason — almost always the ImageVolume feature gate, or a GPU node selector/toleration mismatch. Re-check your actual node labels against inferenceServer.{live,sync}.nodePool.{selector,tolerations}.
  • Predictor pod stuck Init:0/1 — the models-image-tracker initContainer is a trivial command: ["true"] that exists only so ArgoCD's image scanner sees the models image; if it's stuck, the actual model image pull is stuck. Check image.models.tag exists and image.pullSecret can read it.
  • Predictor pod stuck Init:ImagePullBackOff with unable to retrieve auth token: invalid username/password: unauthorized, even though lelapa-ghcr exists and the gateway/triton images pulled fine — you likely only created the secret, not linked it to the pipeline-predictors service account. See the callout in Step 7 above; the model weights volume needs the secret linked at the SA level, pod spec imagePullSecrets alone isn't enough for this volume type on this Kubernetes/CRI-O version.
  • Predictor pod Running but not serving — check the pod-liveness-guard sidecar logs first. It force-deletes the pod after 3 consecutive /v2/health/live failures; if it's cycling, the underlying Triton process is what to debug, not Kubernetes scheduling.
  • sync predictor pod crash-looping with error: creating server: Invalid argument - load failed for model 'infer_mt_nllb_mul_decoder_with_past': ... TensorRT EP could not build execution context, while live loads the identical model fine — GPU VRAM exhaustion on 16GB-class cards (confirmed via nvidia-smi inside the pod: usage hit 14.9GB/16GB immediately before the crash, chart/image 26.8.1). sync's pipeline loads ~14 models (ASR fastconformer, Whisper, diarization/Sortformer, router, langid, NLLB) before it reaches this one, leaving too little headroom for its TensorRT engine build; live only loads ~8 models and has plenty of room for the same model. This is a real capacity issue on Tesla T4 (16GB) specifically, not a config mistake — if you hit it, sync (batch) transcription via /v1/transcribe will be unavailable (503 no healthy upstream) even though the rest of the deployment is healthy. Report it to us; the fix is likely either a larger-VRAM GPU for the sync pool or work on our side to trim/quantize its model set, not something fixable via values tuning on your end.
  • Internal gateway pod not Running / gateway can't reach predictors — inference-router is a plain chart-owned Deployment, no external controller involved, so debug it like any other pod: kubectl get pods -n <namespace> -l app=inference-router, kubectl logs -n <namespace> -l app=inference-router. A crash-looping Envoy container almost always means the rendered bootstrap config is invalid — compare against kubectl get configmap inference-router-config -n <namespace> -o yaml.
  • gateway-secrets missing / pods stuck CreateContainerConfigError — with secrets.mode: existingSecret, the chart never creates this Secret; confirm whatever provisions it (ExternalSecret, SealedSecret, etc.) has actually run. With mode: directValues and CLIENT_USERNAME/CLIENT_PASSWORD left unset, the chart generates random ones on first install — retrieve them with:
    TerminalCode
    kubectl get secret gateway-secrets -n <namespace> -o jsonpath='{.data.CLIENT_USERNAME}' | base64 -d kubectl get secret gateway-secrets -n <namespace> -o jsonpath='{.data.CLIENT_PASSWORD}' | base64 -d
  • gRPC calls to predictors failing / wrong pool — check the internal gateway's rendered bootstrap config (kubectl get configmap inference-router-config -n <namespace> -o jsonpath='{.data.envoy\.yaml}'); sync traffic must carry the x-triton-model: pipeline_sync header or it falls through to the live pool by default.
  • LEAST_REQUEST LB doesn't seem to be doing anything (all traffic hits one pod) — almost always means the predictor Service isn't headless. Confirm kubectl get svc <name>-predictor -n <namespace> -o jsonpath='{.spec.clusterIP}' returns None; otherwise kserve.service.serviceClusterIPNone wasn't set on the KServe installation (see Step 3).
  • Internal gateway routing to a dead/stale predictor pod IP after a rollout — expected, briefly. Backend discovery is DNS-based (internalGateway.dnsRefreshRate, default 5s), not a real control plane watching Endpoints, so a new pod isn't picked up (and a terminated one isn't dropped) until the next DNS refresh. If it's still wrong well past that window, check the predictor Service is actually headless (see the LEAST_REQUEST entry above).
  • HPA not scaling sync — expected. inferenceServer.sync. autoscaling.maxReplicas is pinned to 1 by design.

Qualification

Once every pod is Running/Ready (Step 8 above), confirm the deployment is actually serving correct, on-spec results — not just that the processes are up:

  • Health: curl localhost:9000/health (via the port-forward in Step 8) must return OK with HTTP 200.
  • Predictor readiness: kubectl describe isvc pipeline-live|sync shows a Ready condition of True for each enabled pool.

For a full pass/fail signal — translate/transcribe accuracy against known reference cases, and per-endpoint latency thresholds — run the post-install qualification script from vulavula-examples against your deployment's BASE_URL (e.g. http://localhost:9000 via the same port-forward). It checks:

  1. Sanity — health endpoint, an authenticated translate call, and that /v1/realtime opens a live streaming session.
  2. Accuracy — translation similarity against built-in reference cases (default minimum similarity 0.6); optionally, sync and live transcription Word Error Rate against your own audio sample (default maximum WER 0.36) if you set AUDIO_FILE_PATH / AUDIO_REFERENCE_TEXT.
  3. Performance — per-endpoint latency (defaults: health 1000ms, translate 3000ms, transcribe 15000ms), plus, if you provide an audio sample, live-streaming first-transcript-delta latency (5000ms) and realtime factor (1.5×).

A nominal deployment reports every check as PASS (or SKIP for the optional transcription checks if no audio sample is configured) and exits 0 with an overall QUALIFIED result — thresholds are tunable via .env to whatever envelope is expected for your hardware. See that script's own README.md for setup and full output details.

Need help?

Self-deploying is something we want to support you through, not something to figure out alone. Reach out at [email protected] or join our Discord community and we'll pair you with someone who knows the chart.

Last modified on