Notes
What rendering all 1,481 voices in production found
Every voice in the shop, rendered against production, counted.
Published 2026-08-08 · Last checked 2026-08-14
In short
A full sweep of the catalogue rendered 1,312 of 1,481 voices and failed on 169 - every failure on one engine and clustered in two models. The cause was not the voices: swapping a model while another request was still rendering from it exceeded the container's memory quota and killed every request on it. After the fix, all 1,481 render.
Why sweep at all
Because "the voices work" is a claim, and a claim about 1,481 things needs 1,481 measurements. Spot-checking would have found nothing: the failures were entirely within two models, so any sample that missed them would have looked perfect.
What the numbers said
| Before | After | |
|---|---|---|
| Voices rendering | 1,312 | 1,481 |
| Failures | 169 | 0 |
| Failure mode | "took too long to answer" | - |
| Clustered in | two models, 100% of each | - |
The actual cause
A request for a voice whose model was not loaded evicted a model another request was mid-render on. That pushed the container past its memory quota, the platform killed it, and every in-flight render on it died - including every remaining voice of that model, which is why the failures came in perfect clusters.
The fix is boring and correct: a model in use cannot be evicted, and work queues within what the container's measured memory actually fits. There is a regression test, so the cluster pattern cannot come back quietly.
What we would tell anyone self-hosting
- Size concurrency from measured RSS per render, not CPU count.
- Never evict a model that has an in-flight request.
- Sweep the whole catalogue after any engine change; clusters hide from samples.