Notes
A cold container deleted an entire language from the shop
One shared cache, one silent engine, one missing tier.
Published 2026-08-09 · Last checked 2026-08-14
In short
The app cached the served voice catalogue once, across engines. When one engine was briefly cold after a deploy, the cache kept only the engines that answered - so every voice from the silent one disappeared for five minutes and a saved voice became "unknown voice". Caching per engine, with the last known catalogue retained, fixed it.
Why it looked like a voice bug
Because the symptom was a specific voice failing, reported by the person using it. Nothing in that report points at a cache, and the voice worked again by the time anyone looked - which is the signature of a cache with a short life and a wrong shape.
The shape of the mistake
The catalogue shown in the shop was assembled by asking every engine what it could speak and caching the answer as one list. That is fine while every engine answers. The moment one does not, the union of the successful replies is a smaller, perfectly valid-looking catalogue - and it gets cached as though it were the truth. A voice absent from the catalogue is not a slow voice; it is an unknown one, which is why a saved project failed with a server error rather than waiting.
The cold start itself was ordinary: a container restarted after a deploy and took a few minutes to load its models. Nothing was broken. The cache turned a routine restart into a five-minute outage for a whole language tier.
The rule we now follow
A cache may make things stale. It may not make things disappear. Each engine caches separately and keeps its last known list - falling back to the labelled catalogue on a cold start, since that file ships with the code and cannot be unavailable - so a busy or cold engine can slow the shop but cannot empty a shelf.
There is a regression test for it now, which is the only part of this that stops it happening again: the fix is one function, the reasoning is a paragraph, and the paragraph is what gets forgotten.