Build a RAG Chatbot Over Your Docs with Claude and pgvector

By the end of this you'll have a chatbot that answers from your own documents and tells you which one each answer came from. No managed vector store. No framework papering over the moving parts. Postgres with pgvector, an embedding model, Claude. That's the whole stack.

I've shipped this shape enough times to have opinions about the parts that bite you three weeks later, once there's real data in the table and someone's actually asking it things. Those live in a gotchas section near the end. Read it before you push to prod, seriously.

Audience: you write TypeScript, you've stood up a Next.js app, App Router and all. Embeddings and retrieval, though? No prior exposure assumed. I'll unpack the concepts as we trip over them. I will not stop to explain what a route handler is.

The shape of it, end to end. You slice documents into chunks. Each chunk becomes a vector via an embedding model. Vectors go into Postgres. A question arrives, you embed it the exact same way, ask Postgres which stored vectors sit nearest, and pass those chunks to Claude as context. Claude answers and cites the chunks it leaned on. Five moving parts, and the database does four of them.

What you'll need

• Node 20 or later and a Next.js app (App Router). If you're starting fresh, npx create-next-app@latest.
• Postgres 14+ with the pgvector extension. I'll show you a Docker one-liner.
• An Anthropic API key for Claude.
• An OpenAI API key for embeddings. You can swap the embedding provider later, I'll note where the dimension matters.

Why two providers, briefly. Claude does the answering, the reasoning, the prose, the citations. Embedding is a different job, handled by a smaller and much cheaper model whose entire purpose is turning text into vectors so we can measure how "close" a question is to a chunk. Anthropic doesn't ship a standalone embeddings endpoint, so I lean on OpenAI's text-embedding-3-small for that piece. Swap it for anything you like later. Two rules, both non-negotiable: documents and queries get embedded by the same model, and your Postgres column has to match its output dimension exactly. Break either and your distances turn to mush.

Install the packages:

npm install @anthropic-ai/sdk openai pg
npm install -D @types/pg

And set your environment variables in .env.local:

ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
DATABASE_URL=postgres://postgres:postgres@localhost:5432/ragdemo

Step 1: Postgres with pgvector

The fastest way to get a Postgres that already has pgvector compiled in is the official image:

docker run -d \
  --name rag-postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=ragdemo \
  -p 5432:5432 \
  pgvector/pg17

That image is vanilla Postgres 17 with the extension sitting on disk, present, but inert. Nothing happens until you flip it on. Connect and run:

CREATE EXTENSION IF NOT EXISTS vector;

What pgvector buys you: a vector column type plus a handful of distance operators, so Postgres can answer "which rows point most like this one?" That's it. That's the whole pitch. The usual alternative is standing up a dedicated vector database alongside your real one, another service to provision, monitor, pay for, and page someone about at 3am. For most document-Q&A, that's a service you simply don't need. The database you've already got handles it.

Step 2: The documents and embeddings tables

Two tables. One for source documents, so you've got something concrete to cite back to. One for the chunks and their vectors.

CREATE TABLE documents (
  id          BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  title       TEXT NOT NULL,
  source_url  TEXT,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE chunks (
  id           BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  document_id  BIGINT NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
  chunk_index  INT NOT NULL,
  content      TEXT NOT NULL,
  embedding    vector(1536) NOT NULL
);

That vector(1536) is load-bearing. 1536 is exactly what text-embedding-3-small emits. Move to a model that outputs 768 dims, or 3072, and this number moves with it, no exceptions. Get it wrong and the insert dies with something like expected 1536 dimensions, not 768, which is at least a refreshingly honest error message. Think of the column width and the model's output width as a single fact you've been forced to write down in two places. Postgres polices the agreement; it will not let you smuggle a mismatch through.

ON DELETE CASCADE wipes a document's chunks when the document goes. You'll be grateful for it the first time a source doc changes and you re-ingest, without the cascade you'd be leaving orphaned chunks behind, quietly poisoning retrieval.

Now, the index. Skip it and every similarity search becomes a sequential scan: read every row, compute distance against all of them, sort. Totally fine at a hundred chunks. Genuinely painful at a hundred thousand.

CREATE INDEX ON chunks
  USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 100);

Here's what ivfflat does. It carves your vectors into lists buckets, and at query time it only opens the buckets sitting nearest your query vector instead of scanning the lot. That's the trade, it's an approximate index, giving up a sliver of recall in exchange for a large amount of speed. lists = 100 is a sane place to start; the pgvector docs hand you a rule of thumb of roughly rows / 1000 up to a million rows, and sqrt(rows) beyond that. Now the catch, and it gets people every single time. ivfflat learns its bucket boundaries from whatever rows happen to exist the moment you run CREATE INDEX. Build it on an empty table and the buckets are drawn from nothing, so they partition nothing useful. Load a representative slug of data first, then build the index. Order matters here in a way it doesn't with a plain btree.

Step 3: Chunking

Try embedding a 40-page document as one vector and see how retrieval goes. Badly, is how it goes. A single vector for the whole thing is an average of everything the document says, which means it's about nothing in particular, a smear across every topic at once. So you cut the document into chunks small enough that each one is about a single thing, and you embed those.

Below is a chunker. It splits on paragraph boundaries, packs paragraphs up to a character budget, and carries a little overlap from one chunk into the next:

// lib/chunk.ts
export interface Chunk {
  index: number;
  content: string;
}

export function chunkText(
  text: string,
  { maxChars = 1200, overlapChars = 200 }: { maxChars?: number; overlapChars?: number } = {},
): Chunk[] {
  const paragraphs = text
    .split(/\n\s*\n/)
    .map((p) => p.trim())
    .filter(Boolean);

  const chunks: Chunk[] = [];
  let buffer = "";

  const flush = () => {
    if (!buffer) return;
    chunks.push({ index: chunks.length, content: buffer });
    // Carry the tail of this chunk into the next one for context overlap.
    buffer = overlapChars > 0 ? buffer.slice(-overlapChars) : "";
  };

  for (const para of paragraphs) {
    if (buffer && buffer.length + para.length + 2 > maxChars) {
      flush();
    }
    buffer = buffer ? `${buffer}\n\n${para}` : para;
  }
  flush();

  return chunks;
}

The overlap pulls more weight than it looks like it should. Picture this: the sentence that defines a term sits at the tail of chunk N, and the sentence that uses that term opens chunk N+1. No overlap, and retrieval happily surfaces the chunk that uses the term while missing the one that explains what it means. The answer is half there. Carrying the trailing couple hundred characters forward hedges against exactly that split. Not free, overlapping text gets embedded twice and stored twice, but as insurance goes it's dirt cheap.

I split on blank lines because that's where prose and Markdown draw their real seams. Feed this code, transcripts, or tables and you'll want a splitter that respects those structures, splitting a function in half mid-body is worse than useless. Chunking is the quiet accuracy killer of RAG: the naive default never throws an error, it just slowly makes every answer a little worse. Tune it to your actual content. This bit me once on a corpus that was mostly tables, and I spent a day blaming the embeddings.

Step 4: Generating embeddings

A thin wrapper over the embedding API. Hand it a batch of strings, get a batch of vectors back, same order in as out.

// lib/embed.ts
import OpenAI from "openai";

const openai = new OpenAI();

export const EMBEDDING_MODEL = "text-embedding-3-small";
export const EMBEDDING_DIM = 1536; // must match vector(1536) in the schema

export async function embed(texts: string[]): Promise<number[][]> {
  const res = await openai.embeddings.create({
    model: EMBEDDING_MODEL,
    input: texts,
  });
  // The API preserves input order, but sort by index to be safe.
  return res.data
    .sort((a, b) => a.index - b.index)
    .map((d) => d.embedding);
}

The thing to actually get right here is batching. Embedding endpoints happily take many strings per call, and one round trip for fifty chunks beats fifty round trips on both latency and cost, it's not close. EMBEDDING_DIM sits right next to the model name on purpose. The day someone swaps the model, the dimension is staring at them on the next line, begging to be changed too. And it's the same 1536 that has to match your vector(1536) column. One fact, written down where you'll see both copies at once.

Step 5: Ingesting a document

Time to wire chunking and embedding together into one function: take a document, write the whole thing to Postgres. One wrinkle, pgvector wants the vector as a bracketed string literal, [0.1, 0.2, ...], not a JS array, so there's a tiny formatting step to bridge that.

// lib/ingest.ts
import { Pool } from "pg";
import { chunkText } from "./chunk";
import { embed } from "./embed";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

// pgvector accepts a bracketed, comma-separated string for a vector value.
function toVectorLiteral(vec: number[]): string {
  return `[${vec.join(", ")}]`;
}

export async function ingestDocument(input: {
  title: string;
  sourceUrl?: string;
  text: string;
}): Promise<{ documentId: number; chunkCount: number }> {
  const chunks = chunkText(input.text);
  if (chunks.length === 0) {
    throw new Error("Document produced no chunks.");
  }

  const vectors = await embed(chunks.map((c) => c.content));

  const client = await pool.connect();
  try {
    await client.query("BEGIN");

    const docRes = await client.query<{ id: number }>(
      `INSERT INTO documents (title, source_url) VALUES ($1, $2) RETURNING id`,
      [input.title, input.sourceUrl ?? null],
    );
    const documentId = docRes.rows[0].id;

    for (let i = 0; i < chunks.length; i++) {
      await client.query(
        `INSERT INTO chunks (document_id, chunk_index, content, embedding)
         VALUES ($1, $2, $3, $4)`,
        [documentId, chunks[i].index, chunks[i].content, toVectorLiteral(vectors[i])],
      );
    }

    await client.query("COMMIT");
    return { documentId, chunkCount: chunks.length };
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

The transaction is not decoration. A document that lands with half its chunks written is worse than a document that never landed at all, retrieval will confidently serve up the chunks that made it in and silently skip the ones that didn't, and you'll have no signal that anything's wrong. The answer just quietly loses a third of its evidence. BEGIN/COMMIT makes it all-or-nothing: the document is either fully there or not there.

Call this from wherever suits, a script, an admin route, a background job. For a first run, a throwaway script over a folder of text files does the job:

// scripts/ingest.ts
import { readFile, readdir } from "node:fs/promises";
import { join } from "node:path";
import { ingestDocument } from "../lib/ingest";

const DOCS_DIR = "./docs";

for (const file of await readdir(DOCS_DIR)) {
  if (!file.endsWith(".txt") && !file.endsWith(".md")) continue;
  const text = await readFile(join(DOCS_DIR, file), "utf8");
  const { chunkCount } = await ingestDocument({ title: file, text });
  console.log(`Ingested ${file}: ${chunkCount} chunks`);
}

Step 6: Retrieval

Here's the query that earns its keep. Embed the question, then ask Postgres for the chunks whose embeddings sit closest by cosine distance.

// lib/retrieve.ts
import { Pool } from "pg";
import { embed } from "./embed";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

export interface RetrievedChunk {
  chunkId: number;
  documentId: number;
  title: string;
  content: string;
  distance: number;
}

export async function retrieve(query: string, k = 5): Promise<RetrievedChunk[]> {
  const [queryVec] = await embed([query]);
  const literal = `[${queryVec.join(", ")}]`;

  const res = await pool.query<RetrievedChunk>(
    `SELECT
        c.id          AS "chunkId",
        c.document_id AS "documentId",
        d.title       AS "title",
        c.content     AS "content",
        c.embedding <=> $1 AS "distance"
     FROM chunks c
     JOIN documents d ON d.id = c.document_id
     ORDER BY c.embedding <=> $1
     LIMIT $2`,
    [literal, k],
  );

  return res.rows;
}

That <=> operator is cosine distance. Smaller is more similar; 0 means the two vectors point in identical directions; bigger means further apart. Cosine looks only at direction and throws away magnitude, which is exactly what you want for text, where a long document and a short one about the same topic should read as close. pgvector also ships <-> for L2 (Euclidean) and <#> for inner product. Here's the gotcha that costs people a confusing afternoon: the operator has to match the *_ops class you built the index with. We indexed with vector_cosine_ops, so we query with <=>. Mix them, index cosine, query L2, and Postgres won't use the index at all. It'll fall back to a full scan, silently, and you'll wonder why your "indexed" query crawls.

The ORDER BY ... LIMIT k is the magic phrasing that lets the planner reach for ivfflat in the first place, answering "give me the k nearest" is the one thing that index exists to do. k = 5 is a fine default. Crank it up and Claude gets more to work with, sure, but you're also paying for more input tokens on every question and risking dilution: pad the context with marginally-relevant chunk number 18 and the signal gets noisier, not better.

Step 7: Asking Claude, with citations

Now we hand the retrieved chunks to Claude, ask it to answer from those chunks alone, and ask it to cite them. The trick that makes citations actually reliable, rather than Claude inventing plausible-looking source numbers, is to number each chunk in the prompt and tell it to reference those numbers. We hold the mapping from number to real document on our side and stitch them back together after.

// lib/answer.ts
import Anthropic from "@anthropic-ai/sdk";
import { retrieve, type RetrievedChunk } from "./retrieve";

const client = new Anthropic();

export interface Answer {
  text: string;
  sources: { ref: number; title: string; chunkId: number }[];
}

function buildContext(chunks: RetrievedChunk[]): string {
  return chunks
    .map((c, i) => `[${i + 1}] (from "${c.title}")\n${c.content}`)
    .join("\n\n---\n\n");
}

export async function answerQuestion(question: string): Promise<Answer> {
  const chunks = await retrieve(question, 5);

  if (chunks.length === 0) {
    return { text: "I don't have any documents to answer from yet.", sources: [] };
  }

  const context = buildContext(chunks);

  const system =
    "You answer questions using only the numbered context provided. " +
    "Cite the sources you use with bracketed numbers like [1] or [2], placed " +
    "right after the claim they support. If the context does not contain the " +
    "answer, say so plainly instead of guessing.";

  const msg = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    system,
    messages: [
      {
        role: "user",
        content: `Context:\n\n${context}\n\n---\n\nQuestion: ${question}`,
      },
    ],
  });

  const text = msg.content[0].type === "text" ? msg.content[0].text : "";

  const sources = chunks.map((c, i) => ({
    ref: i + 1,
    title: c.title,
    chunkId: c.chunkId,
  }));

  return { text, sources };
}

That system prompt pulls double duty. "Use only the numbered context" keeps Claude grounded in your documents rather than reaching into its own training, the difference between a RAG system and a chatbot wearing a RAG costume. "If the context doesn't contain the answer, say so" stops it from confabulating one when retrieval comes back empty-handed, which it will, eventually, on some question you didn't anticipate. A confident wrong answer is the worst outcome here. Worse than a crash, worse than a blank. "I don't know" is a feature.

A few details worth dwelling on. The model id is claude-sonnet-4-6, the current Sonnet, fast, cheap enough to run on every question, plenty sharp for grounded Q&A. max_tokens caps response length. The response arrives as a list of content blocks, which is why we check msg.content[0].type === "text" before touching .text, reach for .text blind and it'll bite you the first day a non-text block shows up in slot zero. And we ship the source list back alongside the answer text so the UI can turn those [1], [2] markers into real links pointing at the documents they came from.

Step 8: The API route

Tie the whole thing off behind a Next.js route handler.

// app/api/chat/route.ts
import { NextResponse } from "next/server";
import { answerQuestion } from "@/lib/answer";

export async function POST(req: Request) {
  const { question } = await req.json();

  if (typeof question !== "string" || question.trim().length === 0) {
    return NextResponse.json({ error: "Missing question." }, { status: 400 });
  }

  const answer = await answerQuestion(question);
  return NextResponse.json(answer);
}

Hit it and back comes the answer text plus a sources array you can render however you please:

curl -s localhost:3000/api/chat \
  -H 'content-type: application/json' \
  -d '{"question":"What is the refund window?"}'

{
  "text": "The refund window is 30 days from purchase [1]. Refunds are issued to the original payment method [2].",
  "sources": [
    { "ref": 1, "title": "refund-policy.md", "chunkId": 412 },
    { "ref": 2, "title": "refund-policy.md", "chunkId": 413 }
  ]
}

And that's a working RAG chatbot. Ingest your documents, hit the route, and answers come back with citations that point at the exact chunks they're built on. The happy path is done. Now the part nobody warns you about.

Gotchas

The happy path works. What follows is the stuff I learned the slow way, the failures that don't show up in the demo, only three weeks in.

Chunking is where accuracy is won or lost

Our chunker splits on blank lines, which is genuinely fine for prose. But chunk size is a real lever, not a default to leave alone and forget. Too large, and each vector blurs across several topics, retrieval goes fuzzy, returning chunks that are sort-of-about everything and precisely-about nothing. Too small, and a chunk gets severed from the context that made it mean anything. Retrieval feeling off? This is the first knob to reach for, before you go blaming the model or the index. And again: structured content, code, tables, transcripts, wants a splitter that honors its real boundaries, not blank lines.

Approximate indexes miss things

ivfflat is approximate, and "approximate" is doing real work in that sentence. It only opens the buckets closest to your query, so now and then the true nearest neighbor is hiding in a bucket it never looked in, and your result is a touch worse than an exhaustive scan would've given. Widen the net with SET ivfflat.probes = 10;, more buckets searched, slower queries, better recall. It's a dial, not a switch. If recall genuinely matters and your dataset fits in memory, hnsw is the other index type: better recall, at the cost of slower builds and a hungrier memory footprint. Don't bother reaching for hnsw until ivfflat's recall actually disappoints you on real queries, premature index tuning is a great way to lose an afternoon. And, once more, with feeling: build the index after loading data. ivfflat draws its buckets from whatever rows are present at build time.

Cost adds up in two places

Two meters are running. Ingestion: every chunk costs one embedding call's worth of tokens, and you pay all over again if you ever re-embed, new model, new chunking strategy, doesn't matter. Query time: every question pays for the query embedding plus the Claude call, and that Claude call's input carries all k chunks you crammed into the prompt. So bumping k from 5 to 20 quadruples the context you're billed for, on every question, forever. Embeddings are cheap. Claude input tokens at scale are not. Choose k like it costs money, because it does.

Retrieval can come back empty or wrong, and the model will cover for you if you let it

Here's the real failure mode of RAG, and it isn't a stack trace. It's a fluent, confident, well-cited answer built on chunks that don't actually contain what was asked. It looks right. That's what makes it dangerous. Two defenses, use both. One: that system-prompt instruction to say "I don't know" when the context doesn't cover the question, keep it, and actually test that it fires, because an untested guardrail is just a comment. Two: you're holding the cosine distances already. If the nearest chunk's distance is past some threshold you've calibrated, treat retrieval as a miss and short-circuit before you ever call Claude, cheaper, faster, and honest. Never assume "documents exist" implies "the answer is in them." Those are different claims.

Keep your embeddings consistent

Documents and queries get embedded by the same model. Full stop. The day you upgrade the embedding model is the day every stored vector is suddenly from the old model while your fresh query vectors live in a different space, and cross-space distances are noise dressed up as numbers. Nothing errors. Retrieval just quietly degrades. So: when you change models, re-embed everything, no shortcuts. And pin the model name and dimension in one spot (those embed.ts constants) so the migration is a single obvious edit instead of a scavenger hunt across the codebase at 11pm.

Wrapping up

The core of RAG is far smaller than the ecosystem around it would have you believe: chunk, embed, store, retrieve by distance, answer with the retrieved text as context. Five verbs. Postgres and pgvector cover storage and search with no extra service to babysit, and Claude handles the answer, with citations falling out almost for free, just from numbering the context and asking nicely.

Where I'd point you next. Stream the response, so the answer paints in as it's written instead of landing all at once after an awkward pause. Add metadata filters to the retrieval query, date, source, author, so you can scope which documents are even eligible to be searched. And the big one: stand up a real eval harness. A set of questions with known-good answers, run on every change. Without it, tweaking the chunker or nudging k is pure vibes, you change something, the demo still works, you ship, and you have no idea whether retrieval actually got better or you just got lucky. That harness is the whole difference between tuning RAG and flailing at it. Build it before you think you need it.