Q: What is a vector database, and when do you actually need one?

A vector database stores chunks of your documents, and next to each chunk it stores a vector embedding: a numerical representation of the meaning of that chunk's text. Chunk and vector sit side by side in the database.

Why store them this way? Because of how search works. When a query comes in, you don't compare the text of the query against the text of every chunk. Comparing text against text is slow. Instead, you turn the query into its own vector of meaning, and compare that vector against the chunk vectors. Comparing vectors is a mathematical operation, and it is extremely fast, even across millions of chunks. The database finds the closest vectors and returns the chunks stored next to them. Those chunks are your relevant context.

You don't need to know how the math works. The vector database handles all of it. What you need is the concept: chunks stored with their vectors of meaning, and search turned into a fast mathematical comparison.

So when do you actually need one? When your reference material no longer fits in the prompt. If your material is under roughly 50 pages, you often don't need one yet. Modern context windows can hold all of it, and putting it straight in the prompt is simpler than building retrieval. Past that point, cost, speed, and accuracy all get worse, and retrieval wins.

Practical starting point: use ChromaDB locally while you learn and prototype. ChromaDB is a free, open source vector database. It runs on your own machine and takes minutes to set up, which makes it great for learning these concepts. When you go to production, switch to a managed option. Everything you learned transfers.