What We Self-Host and What We Buy

Self-hosting gets argued as a position. You are either the kind of team that runs its own models or the kind that calls an API, and the discussion is really about identity — control and privacy on one side, focus and velocity on the other.

Treated that way it produces a bad answer, because a retrieval system has several components with completely different economics. We run some of ours and buy others, and the reasoning is the same each time.

The test: has this requirement stopped moving?

Building is not hard. You can build almost anything. The question worth asking is whether the thing you build will still be the thing you need in a year, because the cost of owning software is not the writing of it. It is the decade of changes afterwards.

So the test is: is this requirement small, well understood, and finished moving? If yes, owning it is cheap and the ongoing cost is close to zero. If no, you are signing up to chase someone else's roadmap with a fraction of their team.

We applied that to monitoring once. We were paying for a tool that did about three things we needed and thirty we did not, so we built the three. That was the right call for one reason only: nobody was going to ask for the other thirty, because we already knew we did not want them. The requirement was genuinely finished. Had it been growing, buying would have been obviously correct.

Embeddings: self-hosted, because the volume is enormous and the target is still

Every document you index needs an embedding. Every query needs one. Re-index a corpus and you need all of them again. The call volume is not comparable to generation — it is orders of magnitude higher and it is almost entirely predictable.

It is also a requirement that has stopped moving. An embedding model turns text into a vector of a fixed size. It does not need features. A good one from a year ago is still a good one, and there is no roadmap to keep up with.

High volume, fixed interface, no feature pressure. That is the profile where owning it pays, and it is why the embedding model runs on our own machine.

Reranking: self-hosted, for the same reason and one more

A cross-encoder reranker has the same shape — stable interface, no feature roadmap, called on every single query. It also has to be fast, because it sits directly in the path between a user pressing enter and seeing anything at all.

Adding a network round trip to an external service inside that window is a latency cost paid on every query for the life of the product. Since the GPU is already there for embeddings, the reranker runs beside it and the marginal cost is close to nothing.

The vector database: self-hosted, because the data is the product

This one is less about economics. An index of everything a business knows, in a form designed for retrieval, is an unusually concentrated asset. Keeping it on infrastructure we control is a deliberate choice about where that sits.

It is not a free choice. It means we own capacity planning, memory pressure and upgrades — the work described in what running AI in production actually costs. That is a real ongoing burden and it belongs in the estimate.

Generation: bought, and we expect that to stay true

The model that writes the final answer is the one component we do not run, and it is the one people assume we would want to most.

It fails the test in every direction. The requirement has not stopped moving — capability changes materially on a timescale of months, and a self-hosted choice is a bet that ages. Demand is spiky rather than predictable, which is exactly the shape an API absorbs well and fixed hardware absorbs badly. And the quality gap between a frontier hosted model and what fits comfortably on one machine is still wide at the step where a mistake is most visible to a user.

The counter-argument is confidentiality, and it is a serious one. Where material genuinely cannot leave an environment, that constraint outranks the economics and you self-host generation knowing the trade. What should not happen is choosing it on instinct and discovering the cost afterwards.

The pattern underneath

Read down the table and the split is not ideological. The components we own are the high-volume, stable, boring ones. The component we rent is the one still changing fastest and hardest to keep up with.

That is the general rule, and it survives outside AI: own the parts that have stopped changing and that you use constantly; rent the parts that are still moving. A team that self-hosts everything is paying to maintain someone else's finished work. A team that rents everything is paying a margin on its own most predictable costs.