EngineeringEngineering note
Why Our Chatbot Was Slow: a 12-Second Search Brought Down to 0.07 Seconds on the Same Server
A status page saying the index is healthy only means it is built. Whether the server can serve it fast depends on whether the part searched on every question fits in memory. Ours did not, and nothing in the dashboard said so.
Author
DueClix Engineering
Published
Reading time
5 min read
In May 2026 the chatbot we run on our own server, which answers tax and small-business questions from about 300,000 documents, became painfully slow. For a broad question such as "what is an LLC", only the step that searches the documents took 21 seconds, and still 12.5 seconds when we asked again. The chat window gives up after 15 seconds, so visitors saw "Request timeout" and the question was sent again.
Narrow questions with an exact keyword, such as a filing deadline, came back in under a second. Only the questions that needed a meaning-based search were slow. That split turned out to be the clue.
The index said it was healthy
The vector database reported everything as fine: the index was fully built and the optimiser had nothing left to do. That status only means the search structure exists. It says nothing about whether the server can serve it quickly.
So we stopped reading the status and measured the machine instead:
| What we measured | Value |
|---|---|
| Memory in the server | 15 GB |
| Document pieces stored for search | 8.19 million, 1,024 numbers each |
| Size at full precision | about 34 GB |
| Size of the compressed copy used for searching | about 8.4 GB, plus the search graph |
| Database memory pushed out to disk (swap) | 14.6 GB |
| Disk busy | 97.5%, about 3,000 reads a second |
The part of the data touched on every search was about three times the size of the memory. The operating system kept moving the database's memory to disk and back, so each search turned into thousands of small disk reads on a disk that was already almost fully busy. Keyword searches skip that part, which is why they stayed fast.
Three ways out
- A bigger server. The honest long-term fix, but more cost every month.
- A restart. Clears the swap for a while. It fills again, because the data is still bigger than the memory.
- Make the searched part fit in memory. No new cost, if it can be done safely.
We tried the third first, in two steps.
What we changed
- Removed what nothing could use. About 758,000 stored pieces pointed to documents that no longer existed, or came from bulk archives far outside what the chatbot is for. We deleted them and kept a full list, so any of them can be added again. 8.19 million became 7.44 million.
- Kept the compressed copy in memory. This was the real find. The collection already had a compressed copy of every vector (one byte per number instead of four), but it was set to live on disk. One setting moved it into memory. No rebuilding of the data was needed.
- Let the database tidy itself in a quiet window. One clean-up pass, about three and a half hours, with the chatbot paused while nobody was using it.
| Measure | Before | After |
|---|---|---|
| Search time, warm | 12 to 21 seconds | 0.07 seconds |
| First search after a restart | about 60 seconds | 6.3 seconds |
| Stored pieces | 8.19 million | 7.44 million |
| Extra cost | none |
Compressing vectors can lower answer quality, so we checked six real questions through the full chatbot afterwards (an LLC, a single-member LLC, the 1099 deadline, an EIN for a sole proprietor, S versus C corporation, Texas sales tax). All six answers were correct and quoted the right sources. The reranker, which reorders the results with the full text, makes up for most of what compression loses.
Fast search showed the next bottleneck
With search at 0.07 seconds, the whole answer still took 7 to 34 seconds. That time had always been there, in the reranker and the language model writing the answer. It was simply hidden behind the slow search. Measure each stage separately, or you fix one and are surprised by the next.
What we got wrong afterwards
Two days later, on 21 May, we went further: we also moved the search graph into memory and stopped the operating system from swapping the database at all. Search was fast, but the database now held about 14.6 GB of a 15 GB machine. On 29 May the service that turns each question into numbers (the embedding model) needed memory to restart, could not get it, and was shut down by the system. The chat stopped answering completely. A driver update waiting for a restart made it harder to bring back. It took about 40 minutes and, in the end, removing the swap limit.
The lesson: on a server shared by several services, giving one of them all the memory moves the problem, it does not remove it.
There was a quieter cost too. In July a user told us the chatbot gave vague answers about files that had just been uploaded. The files were fine. The meaning-based search had a 5-second limit, the swapped graph made some searches slower than that, and the code treated a timeout as zero results. The chatbot then answered from the keyword search alone, without the right document. The same question went wrong two times out of three. We raised the limit to 15 seconds the same day, and on 30 July moved the search graph back to disk while keeping the compressed vectors in memory. After that change, warm searches measured 46 to 49 milliseconds. The first search after a restart takes about 9 seconds, a trade-off we accepted.
If your own chatbot is slow, check in this order
- Time each stage separately: turning the question into numbers, the search, reranking, the language model. Fix the slowest one only.
- Compare keyword questions with broad ones. If only broad ones are slow, it is the vector search.
- Check swap and disk while a slow question runs. Memory pushed to disk and a disk near 100% busy means the data does not fit.
- Work out the size: number of stored pieces times numbers per piece times bytes per number. Compare it with the memory the database actually gets.
- Check whether a compressed copy exists and where it lives. Ours was already there, on disk.
- Delete pieces that point to nothing, and keep a list so they can be restored.
- Make search timeouts visible in your logs, so a slow search never quietly becomes a weak answer.
Long questions and large documents also cost money on every answer. Our free token counter shows how many tokens a prompt uses and what it costs on each model.
If your chatbot is slow or gives answers that come and go, tell us what it does. We will tell you which stage is taking the time and what to fix first.