Stop Treating Vector Databases as the Default Answer for Similarity Search

Somewhere in the last two years, the vector database became the new message queue: a component that engineers reach for reflexively, before they have diagnosed the actual problem. A product manager says "semantic search," and a dedicated vector store appears in the architecture diagram before anyone has asked how many vectors they actually have, how often they change, or what latency they need.

This is a pattern worth resisting. Vector databases are genuinely useful tools. They are not universally correct ones.

The Actual Value Proposition

A dedicated vector database earns its complexity cost in a narrow but real set of conditions. You need approximate nearest-neighbor search at scale — hundreds of millions of vectors or more. You need those results in single-digit milliseconds. You need to update the index incrementally without full rebuilds. You need filtering on metadata alongside the vector similarity score. And you need to do all of this under concurrent load from many readers.

When all of those conditions are true simultaneously, a purpose-built system with a well-tuned HNSW or IVF-PQ index, a purpose-built storage layout, and a query planner that understands vector operations is genuinely hard to beat. The engineering tradeoffs that went into these systems — the graph construction parameters, the quantization strategies, the segment merging logic — represent real, hard-won knowledge about a difficult problem.

The mistake is assuming that because the tool solves a hard problem well, every problem in the vicinity is that hard problem.

Most Workloads Are Not That Workload

Consider the median RAG pipeline being built today. It has somewhere between ten thousand and a few million document chunks. It gets queried a few hundred times a day. The embedding model is called once per query. The retrieval step needs to return results in under a second, not under ten milliseconds. The corpus is largely static, updated in batch once a day at most.

For this workload, a flat brute-force index over a few hundred thousand vectors is fast enough. A single-threaded numpy matrix multiply over a million 1536-dimensional vectors takes well under a second on any modern CPU. There is no approximation error. There is no graph construction cost. There is no operational surface area beyond a file on disk.

Or consider Postgres with pgvector. You already have Postgres. Your documents, metadata, user records, and access control logic are already there. Adding a vector column and an HNSW index on that column gives you semantic search that participates in your existing transactions, your existing backup strategy, your existing monitoring stack. The query planner can combine a vector similarity scan with a WHERE user_id = $1 predicate without you writing a custom post-filter. You get this for the cost of an extension and a migration.

The argument for a standalone vector database in this scenario is not a performance argument. It is a complexity argument dressed up as a performance argument.

The BM25 Question Nobody Asks

There is a deeper issue here that the vector database hype has obscured. For a large class of retrieval problems, keyword search with BM25 or a similar sparse retrieval model still outperforms dense vector retrieval — especially on queries that contain precise technical terms, product names, error codes, or identifiers.

Dense retrieval is genuinely better at capturing semantic similarity when the query and the document use different words to express the same concept. It handles paraphrase well. It handles multilingual queries reasonably well. These are real advantages.

But dense retrieval is also worse at exact match. If a user searches for a specific function name, a specific exception class, or a specific configuration key, the embedding model may distribute that signal across dimensions in ways that hurt precision. BM25 finds the exact string. The vector search finds documents that are "about the same kind of thing," which is not always what you want.

Hybrid retrieval — combining a sparse BM25 score with a dense vector score — is the correct default for most production search systems. The interesting engineering question is how to fuse those scores. Reciprocal rank fusion is simple and works well. Learned sparse models like SPLADE blur the line further. But the point is: before you deploy a vector database, you should ask whether you need dense retrieval at all, or whether a well-tuned Elasticsearch or OpenSearch cluster with BM25 already solves your problem.

The Operational Tax Is Real

Standalone vector databases introduce operational surface area that compounds over time. You now have a second stateful system to back up, monitor, upgrade, and reason about during incidents. You have a synchronization problem: keeping the vector index consistent with your source of truth in Postgres or S3 or wherever your canonical data lives. You have a schema evolution problem: when your embedding model changes — and it will change — you need to re-embed your entire corpus and rebuild your index, which is a non-trivial migration for large datasets.

None of these problems are unsolvable. But they are costs. They are costs that are often not on the whiteboard when the architecture is designed, because the person drawing the diagram is thinking about query latency, not about the 2 AM incident when the vector index diverged from the document store and nobody knows which is authoritative.

The question is not "can we afford a vector database?" The question is "does the benefit of a vector database outweigh these costs for our specific workload?" That is a different question, and it deserves a real answer, not a default.

A Decision Framework That Actually Helps

Here is a rough heuristic that holds up in practice:

• Under one million vectors, static or slow-changing corpus, existing Postgres infrastructure: Use pgvector. The operational simplicity wins decisively.
• Under one million vectors, no existing database, latency-insensitive: Use a flat index in memory. Faiss with IndexFlatL2 or a numpy dot product is not embarrassing. It is correct.
• One million to one hundred million vectors, mixed query patterns, metadata filtering required: Benchmark pgvector with HNSW against a dedicated system. The gap may be smaller than you expect, and the operational simplicity of staying in Postgres may be worth a modest latency penalty.
• Over one hundred million vectors, sub-ten-millisecond p99 latency required, high write throughput: This is where a dedicated vector database earns its place. The engineering tradeoffs in systems built specifically for this problem are meaningful at this scale.

The honest version of this framework is: most teams building RAG pipelines or internal semantic search tools are in the first two buckets. They reach for the fourth-bucket solution because the marketing is loud and the tutorials assume it.

The Broader Pattern

The vector database story is a specific instance of a general failure mode in software architecture: the conflation of "this tool solves a hard problem" with "this tool is the right tool for my problem." Kafka solves hard problems in distributed event streaming. It is also catastrophically over-deployed in systems that would be better served by a Postgres table with a processed_at column. The same pattern repeats.

Senior engineers recognize this pattern by its symptoms: the architecture diagram has a component whose operational cost is not proportional to the problem it is solving. The vector database, in many current deployments, is that component.

The right response is not to dismiss vector databases. It is to apply the same rigor to choosing them that you would apply to any other stateful infrastructure decision: understand the cost, understand the benefit, and make the choice explicitly rather than by default.