kvlm is a CLI that brings up vLLM on GPU clouds, measures it under real load, and recommends one flag change at a time, each backed by a verdict you can read. Production pods are built from the flags that earned their keep. It is designed for a human at the keyboard and for AI agents driving the loop end to end. Supports RunPod, Vast.ai, and Kubernetes.
A single static binary. No runtime to install, on your machine or on the pods it manages.
Six commands cover the whole lifecycle. tune and
apply repeat until the measurement says the serving config
is bound by neither its caps nor KV cache. Only then does
ship build production.
This transcript is from a live tuning session on a single H100 running a
27B model in fp8. One measured flag change, verified in production,
everything torn down afterward. kvlm also verifies the served endpoint
end to end, including Anthropic-style /v1/messages
responses.
$ kvlm doctor --target pod ok: driver and CUDA match, imports, disk $ kvlm up --mode profile pod up, vLLM serving under the profiler $ kvlm tune run1: 1226 tok/s total@32, KV 19% verdict: bound by max-num-seqs (running 32 = cap) recommend: kvlm apply max-num-seqs=64 $ kvlm apply max-num-seqs=64 restart verified, engine re-profiled, serving $ kvlm tune run2 (parent: run1, delta: max-num-seqs=64) pressure@48: waiting 15 -> 0, TTFT 1255 -> 221 ms verdict: bound by neither cap nor KV $ kvlm ship --up production pod up from earned flags --max-num-seqs 64 # measured in run2 $ kvlm down all pods terminated, zero burn
Every tune run probes the server beyond its comfort zone
and records where it is bound. The runs form a revision chain: each one
knows its parent and the single flag that changed, so a regression is
always one diff away from its cause.
| pressure phase, 48 streams | run1 (cap 32) | run2 (cap 64) |
|---|---|---|
| running / waiting | 32 / 15 | 48 / 0 |
| TTFT mean | 1255 ms | 221 ms |
| output rate | 381 tok/s | 414 tok/s |
| KV cache used | 15.3% | 22.9% |
| verdict | bound by max-num-seqs | bound by neither cap nor KV |
GPU pods are billed by the minute and fail in creative ways. kvlm treats both as first-class problems, whoever is driving. Verdicts are plain statements, failures arrive with their evidence inline, and every step is resumable, so the same loop works for an engineer at the keyboard or an AI agent driving it end to end. The session above was driven by an agent.
doctor validates your local setup and the target pod
(driver and CUDA match, imports, disk) before a dollar is billed.
Restarts get a realistic budget, dead processes are detected with log tails inline, and long downloads print progress instead of looking like hangs.
up --resume detects an already-launched server and
waits for it instead of clobbering it. Flaky SSH is tolerated, not
mistaken for death.
Every run records its parent and delta. run show,
run diff, and dash reconstruct how the
config got to where it is.
The primary target today. The full loop, including the session above, runs on RunPod H100s.
Deploy and manage pods on Vast.ai marketplace GPUs.
Run the same loop against GPU nodes in your own cluster.
kvlm is written in Goish Rust, our Go-style runtime for Rust. It compiles to a single static binary, so there is nothing to install on the pods it manages.