Lago
Usage-based billing on open source from end to end. Lago rates and invoices, vBilling meters your tenant clusters and sends Lago the events, and both can run on a laptop next to your tenant clusters.
What vBilling sets up
| In Lago | Created by vBilling |
|---|---|
| Billable metrics | One per vBilling metric (vcluster_gpu_hours, vcluster_cpu_core_hours, ...), summing a field named after it (gpu_hours, cpu_core_hours, ...) |
| Plan | vcluster-standard (billing.defaultPlanCode) with one charge per metric at $0, for you to price |
| Customers | One per tenant: external_id is the tenant ID, with its name, email, currency and metadata |
| Subscriptions | sub-<tenant ID> on the tenant's plan (vbilling.vcluster.com/plan, or the default plan) |
| Events | Batches of up to 100. transaction_id is the vBilling event ID, so a resent event is never counted twice. |
Prices, invoices, taxes and the payment provider stay in Lago.
Run Lago locally
The repository ships a Docker Compose file for Lago 1.17, with the UI on :8081 and the API on :3000 (vBilling's own dashboard uses :8080). Lago signs its tokens with an RSA key: generate one, and never commit it. From the repository root:
git clone https://github.com/vClusterLabs-Experiments/vbilling.git && cd vbilling
openssl genrsa 2048 > deploy/lago/lago_rsa.key
echo "LAGO_RSA_PRIVATE_KEY=$(base64 -i deploy/lago/lago_rsa.key | tr -d '\n')" > deploy/lago/.env
docker compose -f deploy/lago/docker-compose.yml --env-file deploy/lago/.env up -dOpen http://localhost:8081 and sign up to create the organization. Then copy the API key from Developers, API keys into a file (paste it, press Enter, then Ctrl-D):
cat > ./lago-api-keyFor a shared or production setup, run Lago in Kubernetes with its Helm chart (see the Lago docs) and point lago.apiURL at its API Service.
Install vBilling with Lago
kubectl create namespace vbilling-system
kubectl -n vbilling-system create secret generic vbilling-lago --from-file=api-key=./lago-api-key
helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \
--set adapters='{lago}' \
--set lago.apiURL=http://lago-api.lago-system.svc.cluster.local:3000 \
--set lago.existingSecret=vbilling-lago| Helm value | Environment variable | Default | Effect |
|---|---|---|---|
lago.apiURL | LAGO_API_URL | http://lago-api.lago-system.svc.cluster.local:3000 | Lago API, reachable from the vBilling pod. Use http://host.docker.internal:3000 for Lago in Docker on the same machine as a vind control plane cluster. |
lago.existingSecret | Secret with the key api-key. Use it instead of lago.apiKey. | ||
lago.apiKey | LAGO_API_KEY | Lago API key. | |
billing.defaultPlanCode | DEFAULT_PLAN_CODE | vcluster-standard | Plan that vBilling creates, and that tenants without a plan annotation subscribe to. |
billing.currency | BILLING_CURRENCY | USD | Currency of the plan and of tenants without a currency annotation. |
Check that vBilling bootstrapped Lago and subscribed the first tenants:
kubectl -n vbilling-system logs statefulset/vbilling | grep -E '\[(bootstrap|lago)\]'
[bootstrap] created metric "vcluster_gpu_hours" (id=...)
[bootstrap] created plan "vcluster-standard" (id=...) with ... charges
[lago] subscribed tenant vcluster-team-a-team-a to plan vcluster-standard
curl -s http://localhost:3000/api/v1/customers -H "Authorization: Bearer $(cat ./lago-api-key)" | jq '.customers[].external_id'Configure pricing
Every charge on vcluster-standard starts at $0, so invoices stay at $0.00 until you price them. In the Lago UI, open Plans, vCluster Standard, and set a price per unit on each charge. vBilling sends every dimension as an event property, so charge filters can price one metric several ways:
sku(orgpu_type): H100 hours differently from L40S hourscapacity_type: a discount forspotandpreemptible, or areservedrateregion,billing_mode,tenant_classandtenant_cluster
To price tenants differently, create more plans and set vbilling.vcluster.com/plan on their tenant clusters. Lago pricing lists the charges to configure, with example price lists.
Events in Lago
Each event carries the quantity under the metric's field name, which the billable metric sums, and every dimension as a property (trimmed here):
{
"transaction_id": "vb1_3f9c...",
"external_subscription_id": "sub-vcluster-team-a-team-a",
"code": "vcluster_gpu_hours",
"timestamp": 1791216000,
"properties": {
"gpu_hours": 0.25,
"sku": "NVIDIA-H100-80GB-HBM3",
"capacity_type": "on-demand",
"billing_mode": "shared",
"region": "us-east-1",
"tenant_cluster": "vcluster-team-a-team-a",
"vcluster": "vcluster-team-a-team-a"
}
}The timestamp is the window start. vcluster repeats tenant_cluster so that charge filters written for v0.1 keep working.
Offboarding
When a tenant has been gone for billing.offboardGrace (default 1h), vBilling terminates its subscription and Lago issues the final invoice.
Good to know
GET /api/v1/reconcilecovers Stripe and Metronome. For Lago, compare the ledger (GET /api/v1/usage) with Lago's current usage for the subscription.- Lago sends no billing signals to vBilling. Set billing states from your own automation with
PUT /api/v1/billing-states/{tenant}, for example{"state": "suspended", "reason": "invoice overdue"}. - Lago also receives
vcluster_gpu_utilization, an informational metric that v0.1 created in Lago and existing plans may still reference. Leave it unpriced.
Run it all on a laptop with vind
vind runs vCluster in Docker, so a control plane cluster, three tenant clusters, vBilling and Lago fit on one machine. Start Lago as above, then, from the repository root:
# a control plane cluster in Docker, with metrics-server
vcluster use driver docker
vcluster create vbilling-host --connect=true
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch deployment metrics-server -n kube-system --type=json \
-p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
# three tenant clusters inside it
vcluster use driver helm
vcluster create team-alpha --namespace vcluster-team-alpha --connect=false
vcluster create team-beta --namespace vcluster-team-beta --connect=false
vcluster create team-gpu --namespace vcluster-team-gpu --connect=false
# vBilling, sending to Lago on the host
kubectl create namespace vbilling-system
kubectl -n vbilling-system create secret generic vbilling-lago --from-file=api-key=./lago-api-key
helm upgrade --install vbilling deploy/helm/vbilling -n vbilling-system \
--set adapters='{lago}' \
--set lago.apiURL=http://host.docker.internal:3000 \
--set lago.existingSecret=vbilling-lagoWithin a minute or two the log shows a window line for the three tenant clusters and a subscription per tenant:
kubectl -n vbilling-system logs statefulset/vbilling
[controller] starting (adapters=[lago], window=1m0s, reconcile=30s, region=default)
API listening on :8080
[controller] window ...: ... events for 3 tenant cluster(s) (...)
[lago] subscribed tenant vcluster-vcluster-team-alpha-team-alpha to plan vcluster-standardOpen the dashboard with kubectl -n vbilling-system port-forward svc/vbilling 8080 and http://localhost:8080/, and the customers in Lago at http://localhost:8081. To clean up:
vcluster use driver docker
vcluster delete vbilling-host
docker compose -f deploy/lago/docker-compose.yml down -v