Copy page

Integrations

Self-host (K8s/Openshift)

What this is

The your-cloud Helm chart deploys the full Lelapa inference stack — the same one powering the managed Vulavula API — into your own Kubernetes cluster. You bring the cluster and GPU capacity; the chart ships everything else needed to run speech-to-text and translation workloads.

It front-loads a small FastAPI gateway (inference-api) that accepts your requests and routes them, via a chart-owned internal Envoy proxy (inference-router), to two GPU-backed Triton predictor pools. The two pools cover the two distinct workloads the API supports: low-latency streaming (live) and higher-throughput batch processing (sync). Models travel as OCI images pulled with the same credentials as the app images — no separate S3 or model-registry plumbing on your side.

This chart does not expose anything outside the cluster or attach authentication — bring your own Ingress / Gateway / LoadBalancer and auth layer in front. See the chart's own README.md, RUNBOOK.md, and DESIGN.md (shipped inside the chart tarball at charts/your-cloud/) for the full, maintained reference. The steps below are a quickstart; reach out to us first (see the callout above) to get access.

Prerequisites

Before you start, confirm with us which your-cloud chart version and which their-cloud-mvp image release tag you should pin — both are private (GHCR internal visibility) and version advice is per-customer.

• Kubernetes ImageVolume feature gate (alpha since 1.31, off by default) — the predictor pods mount model weights straight out of an OCI image; there's no fallback path. Confirm the gate is actually on with whoever manages the cluster.
• At least one GPU in the cluster for inference (plan for 3 GPUs of headroom at default autoscaling limits — live scales up to 2 replicas × 1 GPU, sync is pinned to 1 replica × 1 GPU). Set inferenceServer.{live,sync}.nodePool.{selector,tolerations} in your values file to match your actual GPU node's labels/taints — the chart ships no default here, leaving it unset means predictor pods get no selector/toleration at all.
• NVIDIA drivers + device plugin already on those GPU nodes, so nvidia.com/gpu shows up as allocatable. This is a real, easy-to-miss prerequisite on OpenShift specifically — neither ARO nor (per NVIDIA's own docs) ROSA ships GPU drivers by default, unlike some managed AKS/EKS node images. You'll separately need to install Node Feature Discovery + the NVIDIA GPU Operator from OperatorHub (a Subscription for each, plus a ClusterPolicy CR) before nvidia.com/gpu becomes allocatable at all:

  Code
  kubectl describe node <gpu-node-name> | grep -A5 Allocatable
  # look for nvidia.com/gpu in the output

  Driver installation can trigger a node reboot the first time it loads — expect a NodeNotReady/NetworkPluginNotReady blip on that node for a couple of minutes; it self-heals once the node comes back.
• kubectl or oc, plus helm v3.x, pointed at the target cluster. Every command below is written for kubectl/plain Kubernetes; on OpenShift substitute oc 1:1 for kubectl (e.g. oc get pods instead of kubectl get pods) and additionally grant the restricted-v2 SCC to the service account your pods run as — see the OpenShift callout after Step 7.
• A lelapa-ghcr image pull secret in the target namespace — the app, runtime, and model weights images are all private (GHCR internal). Create it with a GitHub PAT that has read:packages scope (Step 7 below); the chart never provisions it itself.
• Your own Ingress / Gateway + auth in front of the chart's internal :9000 service — the chart does not expose anything outside the cluster, terminate TLS, or attach authentication. Its only built-in auth is a Basic Auth pair on the service itself, not a substitute for real edge auth.

Install

Update

Code
helm diff upgrade your-cloud oci://ghcr.io/lelapa-ai/charts/your-cloud \
  --version <X.Y.Z> \
  -f my-values.yaml -n <namespace>     # requires the helm-diff plugin

helm upgrade --install your-cloud oci://ghcr.io/lelapa-ai/charts/your-cloud \
  --version <X.Y.Z> \
  -f my-values.yaml -n <namespace>

Same command for first install and every subsequent upgrade — helm upgrade --install is idempotent. Bump only the chart --version to move chart templates / schema / defaults; bump only image.tag in your values file to move what code / model weights run. Tell us when you intend to do either so we can confirm version compatibility.

Rollback

Code
helm history your-cloud -n <namespace>
helm rollback your-cloud <REVISION> -n <namespace>

Rollback reverts the chart's rendered manifests to a previous release, not any data — the metering-db PVC (if enabled) isn't touched either way.

Uninstall

Code
helm uninstall your-cloud -n <namespace>

The metering-db-pvc PVC is not deleted automatically by Helm — remove it manually if you actually want the data gone:

Code
kubectl delete pvc metering-db-pvc -n <namespace>

Troubleshooting

• Predictor pod stuck Pending, no obvious reason — almost always the ImageVolume feature gate, or a GPU node selector/toleration mismatch. Re-check your actual node labels against inferenceServer.{live,sync}.nodePool.{selector,tolerations}.
• Predictor pod stuck Init:0/1 — the models-image-tracker initContainer is a trivial command: ["true"] that exists only so ArgoCD's image scanner sees the models image; if it's stuck, the actual model image pull is stuck. Check image.models.tag exists and image.pullSecret can read it.
• Predictor pod stuck Init:ImagePullBackOff with unable to retrieve auth token: invalid username/password: unauthorized, even though lelapa-ghcr exists and the gateway/triton images pulled fine — you likely only created the secret, not linked it to the pipeline-predictors service account. See the callout in Step 7 above; the model weights volume needs the secret linked at the SA level, pod spec imagePullSecrets alone isn't enough for this volume type on this Kubernetes/CRI-O version.
• Predictor pod Running but not serving — check the pod-liveness-guard sidecar logs first. It force-deletes the pod after 3 consecutive /v2/health/live failures; if it's cycling, the underlying Triton process is what to debug, not Kubernetes scheduling.
• sync predictor pod crash-looping with error: creating server: Invalid argument - load failed for model 'infer_mt_nllb_mul_decoder_with_past': ... TensorRT EP could not build execution context, while live loads the identical model fine — GPU VRAM exhaustion on 16GB-class cards (confirmed via nvidia-smi inside the pod: usage hit 14.9GB/16GB immediately before the crash, chart/image 26.8.1). sync's pipeline loads ~14 models (ASR fastconformer, Whisper, diarization/Sortformer, router, langid, NLLB) before it reaches this one, leaving too little headroom for its TensorRT engine build; live only loads ~8 models and has plenty of room for the same model. This is a real capacity issue on Tesla T4 (16GB) specifically, not a config mistake — if you hit it, sync (batch) transcription via /v1/transcribe will be unavailable (503 no healthy upstream) even though the rest of the deployment is healthy. Report it to us; the fix is likely either a larger-VRAM GPU for the sync pool or work on our side to trim/quantize its model set, not something fixable via values tuning on your end.
• Internal gateway pod not Running / gateway can't reach predictors — inference-router is a plain chart-owned Deployment, no external controller involved, so debug it like any other pod: kubectl get pods -n <namespace> -l app=inference-router, kubectl logs -n <namespace> -l app=inference-router. A crash-looping Envoy container almost always means the rendered bootstrap config is invalid — compare against kubectl get configmap inference-router-config -n <namespace> -o yaml.
• gateway-secrets missing / pods stuck CreateContainerConfigError — with secrets.mode: existingSecret, the chart never creates this Secret; confirm whatever provisions it (ExternalSecret, SealedSecret, etc.) has actually run. With mode: directValues and CLIENT_USERNAME/CLIENT_PASSWORD left unset, the chart generates random ones on first install — retrieve them with:

  Code
  kubectl get secret gateway-secrets -n <namespace> -o jsonpath='{.data.CLIENT_USERNAME}' | base64 -d
  kubectl get secret gateway-secrets -n <namespace> -o jsonpath='{.data.CLIENT_PASSWORD}' | base64 -d

• gRPC calls to predictors failing / wrong pool — check the internal gateway's rendered bootstrap config (kubectl get configmap inference-router-config -n <namespace> -o jsonpath='{.data.envoy\.yaml}'); sync traffic must carry the x-triton-model: pipeline_sync header or it falls through to the live pool by default.
• LEAST_REQUEST LB doesn't seem to be doing anything (all traffic hits one pod) — almost always means the predictor Service isn't headless. Confirm kubectl get svc <name>-predictor -n <namespace> -o jsonpath='{.spec.clusterIP}' returns None; otherwise kserve.service.serviceClusterIPNone wasn't set on the KServe installation (see Step 3).
• Internal gateway routing to a dead/stale predictor pod IP after a rollout — expected, briefly. Backend discovery is DNS-based (internalGateway.dnsRefreshRate, default 5s), not a real control plane watching Endpoints, so a new pod isn't picked up (and a terminated one isn't dropped) until the next DNS refresh. If it's still wrong well past that window, check the predictor Service is actually headless (see the LEAST_REQUEST entry above).
• HPA not scaling sync — expected. inferenceServer.sync. autoscaling.maxReplicas is pinned to 1 by design.

Qualification

Once every pod is Running/Ready (Step 8 above), confirm the deployment is actually serving correct, on-spec results — not just that the processes are up:

• Health: curl localhost:9000/health (via the port-forward in Step 8) must return OK with HTTP 200.
• Predictor readiness: kubectl describe isvc pipeline-live|sync shows a Ready condition of True for each enabled pool.

For a full pass/fail signal — translate/transcribe accuracy against known reference cases, and per-endpoint latency thresholds — run the post-install qualification script from vulavula-examples against your deployment's BASE_URL (e.g. http://localhost:9000 via the same port-forward). It checks:

1. Sanity — health endpoint, an authenticated translate call, and that /v1/realtime opens a live streaming session.
2. Accuracy — translation similarity against built-in reference cases (default minimum similarity 0.6); optionally, sync and live transcription Word Error Rate against your own audio sample (default maximum WER 0.36) if you set AUDIO_FILE_PATH / AUDIO_REFERENCE_TEXT.
3. Performance — per-endpoint latency (defaults: health 1000ms, translate 3000ms, transcribe 15000ms), plus, if you provide an audio sample, live-streaming first-transcript-delta latency (5000ms) and realtime factor (1.5×).

A nominal deployment reports every check as PASS (or SKIP for the optional transcription checks if no audio sample is configured) and exits 0 with an overall QUALIFIED result — thresholds are tunable via .env to whatever envelope is expected for your hardware. See that script's own README.md for setup and full output details.

Need help?

Self-deploying is something we want to support you through, not something to figure out alone. Reach out at [email protected] or join our Discord community and we'll pair you with someone who knows the chart.