What Running AI in Production Actually Costs

Ask what an AI feature costs to run and you will get an answer about tokens. It is the number vendors publish, it scales in an intuitive way, and it fits in a spreadsheet. It is a real cost and it is worth managing.

It has also, in our experience, never been the cost that caused a problem. The thing that puts a retrieval system on the floor is almost always memory — and it does not announce itself as a cost at all. It announces itself as a product that got slow.

The index has to live somewhere

A generation call is a transaction. It happens, you pay for it, it is over. A vector index is a resident. It occupies memory for as long as the product exists, whether anyone is asking it anything or not.

That distinction is the whole cost story. Traffic-driven costs scale with usage and stop when usage stops. The index does not care. It is the same size at three in the morning, and it is sized by how much material you have indexed rather than by how many people are asking.

Which means the growth curve nobody budgets for is content growth, not user growth. Adding documents is the cheap, invisible action that quietly raises your floor.

What running out of memory looks like

It does not look like an out-of-memory crash, which would at least be honest. The operating system starts swapping — moving memory to disk to keep everything nominally running. Nothing fails. Everything survives. And a search that took a fraction of a second starts taking many seconds, because a structure designed for random access in RAM is now being read off a disk.

This is the worst failure mode a system can have, because every dashboard stays green. The service is up. Error rates are normal. The only signal is latency, and latency gets attributed to the model, because the model is the part everyone assumes is slow.

We have watched a retrieval layer degrade to the point of timing out on a large share of queries with no error anywhere in the stack, purely from memory pressure. Nothing in the application had changed. The index had simply outgrown the machine.

Configuration is cheaper than hardware, and you should exhaust it first

The instinct at that point is to buy a bigger machine. Sometimes that is right. It should not be the first move, because the levers inside a vector database are large and cost nothing.

• Quantisation. Storing vectors at reduced precision cuts the memory an index needs by a substantial factor, with a small and measurable effect on result quality. Measure the quality change on your own queries; do not assume it, in either direction.
• On-disk structures. Keeping the graph or the raw vectors on disk while serving from a smaller in-memory working set trades a little latency for a lot of headroom.
• Deleting what you never should have indexed. Crawled duplicates, boilerplate and abandoned uploads are pure cost. They occupy memory, they slow search, and they compete with real content for a place in the results.

That last one is the least glamorous and has repeatedly been the biggest single win for us. An index is not an archive. Material that will never be a good answer to any question is not neutral — it is paid for continuously, and it makes the results worse.

Self-hosted models are not free, they are fixed

Running your own embedding or reranking model removes a per-call charge and replaces it with a machine you pay for continuously. That is a good trade at high volume and a poor one at low volume, and the crossover is a real calculation rather than a matter of principle.

The part that gets missed is that a fixed cost is also a fixed ceiling. An API absorbs a traffic spike and sends you a larger bill. Your own hardware absorbs a spike by getting slower, and then by swapping. Fixed cost means fixed capacity, and capacity is the thing that fails first.

How to budget for this honestly

Cost an AI system the way you would cost a database-backed application, because that is mostly what it is. The model is a line item, not the architecture.

1. Size the index for the content you expect in a year, not the content you have at launch.
2. Establish what retrieval latency looks like on an unloaded machine, so you can recognise degradation rather than argue about it.
3. Alert on memory headroom and swap, not only on errors. This failure never produces an error.
4. Treat indexed volume as a cost centre with an owner, or it will grow without anyone deciding it should.
5. Re-measure after any change to quantisation or storage mode, on your own queries.

None of this is exotic. It is capacity planning, applied to a component that teams are used to thinking of as intelligence rather than as a large data structure with a memory bill. The systems that stay fast are the ones whose owners made that switch early.