Holding Long Conversations in GPU Memory
Capacity engineering for a self-hosted, real-time LLM service on vLLM and Nemotron 3 Nano
Full paper: PDF, 5 pages
Advisory Guide is a customer support workflow tool that gives an agent short, grounded suggestions during a live conversation. The agent loads preparation material (product documentation, account notes, policies), the client transcribes the call locally, and when the user presses Suggest, a GPU worker returns a short answer in a few seconds.
That workload has an unusual shape. Every session keeps a large, stable preparation prefix and a growing transcript resident on the GPU, but only a small fraction of sessions generate at any moment. The serving cost is set by memory, not compute. This post summarizes how we qualified capacity on one H100 PCIe 80GB with vLLM 0.17.
1. Find the constraint that actually binds
Measured preparation bundles span 15K to 49K tokens, and each Suggest appends about 316 tokens, almost perfectly linearly. A thirty-minute call roughly doubles a session. Most of that state is idle at any instant, so two limits act independently: resident context fills the KV pool whether or not anyone is generating, and active requests consume compute.
Before touching hardware, two investigations pointed at the request path. In one, the client waited far longer than the runtime reported spending on GPU work. In the other, latency grew with session count because the session memory rewrote earlier context each turn, invalidating the prefix cache on every request. Making the prompt append-only (stable preparation first, new turns only added at the end) restored reuse on the same GPU.
2. Choose the model for that constraint
We replaced Qwen3.6-27B (a DeltaNet hybrid) with NVIDIA Nemotron 3 Nano 30B-A3B. Most of its layers are Mamba-2 blocks with fixed-size state; the few attention layers use grouped-query attention with two KV heads; and the cache is stored in FP8. Mixture of experts reduces compute per token but not memory, since every expert stays resident, so none of the gain is credited to it.
| Model and cache precision | Reported KV pool |
|---|---|
| Qwen3.6-27B, FP8 weights, 16-bit KV | 183,456 tokens |
| Qwen3.6-27B, FP8 weights, FP8 KV | 374,752 tokens |
| Nemotron 3 Nano 30B-A3B, FP8 weights, FP8 KV | 2,973,696 tokens |
At matched FP8 cache precision that is 7.9x more context per GPU.
3. Confirm it under load: the cache is a step, not a slope
Prefix cache hit rate vs resident tokens
Raising resident context from 0.25x to 2.0x of the pool, prefix hit rate held at about 83% (a structural ceiling: with 4,224-token blocks and ~14,900-token prompts, the trailing partial block is always recomputed). The last clean point was 1.063x at about 81%. The first failing point, 1.197x, measured exactly 0.0%. This is the expected behavior of a least-recently-used cache under a rotating access pattern: once the working set exceeds capacity, each access evicts the entry needed next. What the measurement adds is where that edge falls for this workload, and how close to the pool size it sits.
Three more results shaped admission:
- Rate is a second limit. The gated ceiling was 3.0 requests per second. Raising
max_num_seqsfrom 16 to 64 did not move it, but roughly halved overload latency. At any realistic press cadence, rate binds before residency. - Admission must be a token budget. The same pool holds 198 sessions of 15K tokens or 60 of 49K. And sessions grow: 200 sessions admitted at 0.61x of the pool reach 1.25x after thirty minutes with zero new admissions.
- Output length is capacity. In an earlier run, a harness that omitted the production length instructions produced 169-token answers instead of 36 and moved p90 from 3.21 s to 23.95 s. That run measured a missing prompt, not the GPU.
With admission at or below the pool, a seven-hour soak at 2.75 requests per second served 69,312 requests across 200 resident sessions with 0 errors and 0 preemptions, absorbing 3,440 mid-stream disconnects.
4. Trust, failure, and what remains open
- Health is not capability. One server passed health checks while every first generation failed: a kernel compiled on first use needed a missing build tool. Readiness now requires a real inference and a release-identity check.
- Recoverable is not reproducible. The approved artifact restored from object storage with all 18 file hashes matching, but a fresh export came out byte-different because quantization calibration inputs had drifted.
- Open item. Answer quality with an FP8 KV cache has not yet been re-validated at full context length. All numbers are from one Lambda H100 PCIe and are non-certifying for other hardware.
Takeaway
Find the constraint that binds, act on it with a specific mechanism, confirm it under the real workload, and state the limits plainly. For long-context serving, that means sizing by the KV pool, admitting by a growing token budget, and treating the cache cliff as a wall.