▶️ Watch the video: https://www.youtube.com/watch?v=HlT40sMLex0

Two databases, one sync job

Most RAG tutorials start the same way: documents live in Postgres, and their embeddings go to a dedicated vector database. Two systems means a sync job. Every insert, update and delete has to reach both, in order, or your agent retrieves things that no longer exist.

It also means two security models. Postgres knows this user can only see their own company's contracts. The vector database doesn't, so permissions get copied into metadata filters, and the copies drift. And every question pays two round trips: one to find similar chunks, another to fetch the real rows.

For most enterprise workloads there's a simpler answer: keep the embeddings next to the data they describe.

One database: data and embeddings

One Postgres database holds the documents, the tenants, the permissions and, with the pgvector extension, the embeddings. Same rows, same transactions, same backups.

When a document changes, the change and an outbox event commit together. An embedding worker picks up the event, calls the embedding model and writes the vectors back. It's the same queue-and-worker pattern from the previous article on AI middleware.

Delete a document and its embeddings go with it; there's nothing left to orphan. At query time the RAG API runs one SQL query that applies similarity, business filters and permissions together. Row-level security covers vectors exactly like it covers everything else. The best chunks then go to the LLM as context.

The schema

Store chunks, not one embedding per document. A 40-page contract has to be split, or the one paragraph that matters gets averaged away.

Each chunk references its document with ON DELETE CASCADE. Delete the contract, and every chunk and every vector goes with it, in the same transaction. That's the bug from the opening, gone at the schema level.

Three more details:

• The vector dimension must match your embedding model (1536 here).
• An HNSW index with cosine distance. Without it, every query scans every row.
• Row-level security, so every query on this table, vector search included, only sees the current tenant's rows.

-- schema.sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
  id        bigserial PRIMARY KEY,
  doc_id    uuid NOT NULL REFERENCES enterprise_docs(id) ON DELETE CASCADE,
  tenant_id uuid NOT NULL,
  content   text NOT NULL,
  embedding vector(1536)
);

CREATE INDEX ON doc_chunks USING hnsw (embedding vector_cosine_ops);

ALTER TABLE doc_chunks ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON doc_chunks
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

Retrieval in one query

Embed the question with the same model you used for the documents. Then open a transaction and set the tenant for this request. The true at the end of set_config scopes the setting to the transaction, so a pooled connection never carries one customer's tenant into the next request.

In the query, <=> is cosine distance: order by it and take the top five. Because it's plain SQL, the chunks join the business data in the same statement: only active documents, only chunks that already have an embedding. No second system, no metadata copy.

The last line applies a distance threshold. If nothing is close enough, return nothing. An agent with no context should say "I don't know", not improvise from the fifth-best match.

// retrieve.ts
export async function retrieve(tenantId: string, question: string) {
  const embedding = await embed(question);

  return db.transaction(async (tx) => {
    await tx.query("SELECT set_config('app.tenant_id', $1, true)", [tenantId]);

    const { rows } = await tx.query(
      `SELECT c.content, d.title, c.embedding <=> $1 AS distance
         FROM doc_chunks c
         JOIN enterprise_docs d ON d.id = c.doc_id
        WHERE d.status = 'active' AND c.embedding IS NOT NULL
        ORDER BY c.embedding <=> $1
        LIMIT 5`,
      [toSql(embedding)],
    );

    return rows.filter((r) => r.distance < 0.35);
  });
}

Keeping embeddings fresh

When a document changes, three things happen in one transaction: the new content is saved, the old chunks are removed, and an outbox event is written. Either all of it commits or none of it does. For a few seconds the document has no chunks, so it simply isn't retrieved. Missing context is safe; stale context is not.

The worker does the slow part outside the transaction: split, embed, write back. If the document was deleted in the meantime, there's nothing to do. And the write-back checks the version, so a slow job can never overwrite a newer document with old vectors.

// reindex.ts
// 1. The change and its event commit together
await db.transaction(async (tx) => {
  await tx.query('UPDATE enterprise_docs SET content = $2, updated_at = now() WHERE id = $1',
    [id, content]);
  await tx.query('DELETE FROM doc_chunks WHERE doc_id = $1', [id]);
  await tx.query("INSERT INTO outbox (topic, doc_id) VALUES ('doc.changed', $1)", [id]);
});

// 2. The embedding worker does the slow part outside the transaction
new Worker('embeddings', async (job) => {
  const doc = await docs.get(job.data.docId);
  if (!doc) return; // deleted: its chunks went with it

  const chunks = split(doc.content, { maxTokens: 500, overlap: 50 });
  const vectors = await embedMany(chunks);

  await docs.replaceChunks(doc.id, doc.updatedAt, chunks, vectors); // skipped if the doc changed
});

Where this breaks

Filtered search. HNSW finds the nearest neighbours first and applies your filter after. With a selective filter, like one small tenant in a huge table, you ask for five results and get two. pgvector 0.8 added iterative index scans for exactly this; turn them on:

SET hnsw.iterative_scan = relaxed_order;

Index memory. HNSW is fast when the index fits in RAM. Size the instance for it, and expect index builds on millions of rows to take a while.

Exact identifiers. Ask for contract C-4471 and vector search returns contracts that look similar. Embeddings are bad at codes and SKUs. Postgres already has full-text search, so combine both rankings in one query.

Real scale. For most companies, with thousands to tens of millions of chunks, Postgres is enough. At billions of vectors or thousands of queries per second, a dedicated vector database earns its cost. Choose it for that reason, not by default.

What the business gets

• One system of record for data and embeddings: one backup, one audit trail, and one access model your security team has already reviewed.
• No sync job, so no orphaned vectors and no agent quoting deleted documents.
• One query per question instead of two round trips, and one less vendor to contract, secure and monitor.

You don't need a new database to build RAG. You need the one you already have, used properly.

The numbers in the video are targets for an illustrative scenario, not measurements. The code here is simplified; the full version is in the repository.

▶️ Watch the video: https://www.youtube.com/watch?v=HlT40sMLex0

💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v2-rag-pgvector

🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1

▶️ Watch the video: https://www.youtube.com/watch?v=JQ25ZvyS3Uk

AI Middleware Architecture

The mistake is treating an LLM like a database call. It isn't one. It's slow, it's non-deterministic, and it fails in ways your ERP was never designed to handle. So you never connect them directly. You put a middleware layer in between, and that layer has one job: absorb the chaos.

Step one: the ERP fires a webhook and gets an immediate response. 200 milliseconds, done. The ERP is free.

Step two: the event lands in a queue. Not processed, queued. Now you control throughput, retries, and rate limits against the model provider.

Step three: stateless workers pull jobs and call the model. They scale horizontally: two workers on a quiet Tuesday, 40 on Black Friday.

Step four is the one most teams skip: validation. The model's answer is untrusted input. It gets parsed against a strict schema before it ever touches your ERP. If confidence is low, it goes to a human. Every decision gets written to an audit log. The ERP never waits, and the model never writes directly to your system of record.

Ingestion: ack fast, process later

The ingestion endpoint does three things before it answers.

First, it verifies the webhook signature. This endpoint can trigger financial operations, so unsigned requests die here.

Second, it parses the payload with Zod. Malformed events fail fast, before they cost you a single token.

Third, and this line matters, the job ID is the ERP's event ID. ERPs retry webhooks. If the same order arrives three times, the queue deduplicates it. That's your idempotency, in one parameter.

Then the endpoint returns 202 Accepted, not 200. It tells the ERP: received, not finished.

// webhook.ts
app.post('/erp/events', async (req, res) => {
  if (!verifySignature(req)) return res.sendStatus(401);

  const event = ErpEvent.parse(req.body);

  await queue.add(event.type, event, { jobId: event.id });

  res.sendStatus(202);
});

The AI worker: schema, gate, audit

The worker handles a logistics routing case: given an order, pick the carrier and the warehouse. Notice the schema. The carrier isn't free text; it's an enum. The model cannot invent a carrier that doesn't exist in your contracts. The worker requests structured output from the model, and then parses it again anyway. Trust, but verify.

Then comes the confidence gate. Below 0.85, the decision goes to a human review queue instead of the ERP. You tune that threshold with real data, not intuition. Only then does the worker write back to the ERP and log the decision, including the model's reasoning.

// routing.worker.ts
const RoutingDecision = z.object({
  carrier: z.enum(['DHL', 'FEDEX', 'LOCAL_FLEET']),
  warehouseId: z.string(),
  confidence: z.number().min(0).max(1),
  reason: z.string(),
});

new Worker('erp', async (job) => {
  const order = job.data;
  const decision = RoutingDecision.parse(
    await llm.generate({ schema: RoutingDecision, input: buildContext(order) })
  );

  if (decision.confidence < 0.85) {
    return reviewQueue.add('routing', { order, decision });
  }

  await erp.updateShipment(order.id, decision);
  await audit.log(order.id, decision);
});

What the business gets

• 0 ms AI latency added to the ERP
• 0 Duplicate orders processed
• 100% AI decisions auditable

What does this buy the business?

First, your ERP is isolated from AI latency. Model providers have bad days. Your order pipeline doesn't have to.

Second, idempotency. No order is routed twice. No invoice is flagged twice. In finance and logistics, that's the difference between an automation and an incident.

Third, auditability. When the CFO asks why an invoice was flagged, or why a shipment went to the wrong warehouse, you have the input, the output, the model version, and the reasoning for every decision.

The numbers are targets for an illustrative scenario, not measurements.

Same middleware, new use case

The pattern is also reusable. Swap the schema and the prompt, and the same middleware runs invoice analysis: duplicate charges, pricing mismatches against contracts, anomalies in payment terms. Same queue, same validation, same audit trail.

The model is the least important part of this architecture. The layer around it is what makes it enterprise-grade.

▶️ Watch the video: https://www.youtube.com/watch?v=JQ25ZvyS3Uk

💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v1-ai-middleware

🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1

Routing can cut most of your LLM bill. In this article we build a router that saves money without making this mistake, and the check that catches it when it does.

▶️ Watch the video: https://www.youtube.com/watch?v=7kZ9mOlxuZo

One model for everything

Most teams start with one model for everything, usually the best one, because nobody wants to be blamed for a bad answer. But consider what goes through it: classifying a ticket, pulling the total from an invoice, translating a sentence. These are jobs a model at a tenth or a twentieth of the price does just as well.

You're paying frontier prices for autocomplete. So you route: each request goes to the cheapest model that can handle it. The hard part is "can handle it".

Small, medium or large?

• Classify a ticket: billing, bug or sales: small · one word from a fixed list
• Summarize a 40-page supplier contract: medium · long, but low risk
• Translate "your order has shipped": small
• "Can we end the Globex contract early?": large · 7 words, legal risk

For each of the four requests above, would you pick the small, medium or large model? Decide before reading on.

One: classify a ticket into three labels. Small. The answer is one word from a fixed list, and you can check it.

Two: summarize a 40-page supplier contract. Long, but it's a summary for a person to read. Medium is enough.

Three: translate "your order has shipped". Small, easily.

Four: the 7-word contract question from the introduction. Large. It's the shortest of the four, and the riskiest. A wrong yes has legal consequences. Length tells you what a prompt costs, not how hard it is. Request four comes back in the router code below.

Route first, then validate

Every request goes through a router before any model sees it. The router is not another LLM. It's plain code: the task type, a few risk rules, and a token count. It picks the cheapest tier that's allowed. Then the answer goes through a validator. Does it match the schema? Does it pass the checks for that task? If it fails, the request moves up one tier. That's a cascade: you only pay for the big model when the small one actually failed.

Step by step

The log below traces two requests. Acme sends an invoice. The task rule says small. The small model misses the total, so the schema check fails, and the request moves up to medium, which gets it right. One escalation, still a fraction of the large model's price. Then comes the contract question: the risk rule sees a legal task and goes straight to large. No guessing.

→ POST /ask  task=extract_invoice  tenant=acme
  route ..................... small    (task rule)
  small ....... 0.4 s  $0.0002   total=null       ✗ schema
  escalate .................. medium
  medium ...... 1.1 s  $0.0021   total=18,420.00  ✓
→ POST /ask  task=contract_qa  "Can we end the Globex contract early?"
  route ..................... large    (risk: legal)
2 requests · 1 escalation · 0 wrong answers

The router

The code below implements the router. Each task starts at a tier you tested. Classifying and translating: small. Summaries: medium. A task nobody has mapped yet goes to large. An unknown task should cost more, not risk more. Long inputs get at least medium. This is where request four is handled: anything tagged legal, payment or medical goes straight to large, whatever its length. That is the line the router from the introduction was missing.

// llm/route.ts
const TIERS = ['small', 'medium', 'large'] as const;
type Tier = (typeof TIERS)[number];

// Each task starts at the tier you tested, not the one you guessed
const TASK_TIER: Partial<Record<Task, Tier>> = {
  classify_ticket: 'small',
  translate: 'small',
  extract_invoice: 'small',
  summarize: 'medium',
};

const RISKY = new Set(['legal', 'payment', 'medical']);

export function route(req: LlmRequest): Tier {
  if (req.tags.some((tag) => RISKY.has(tag))) return 'large';

  const tier = TASK_TIER[req.task] ?? 'large';
  return req.inputTokens > 30_000 ? atLeast(tier, 'medium') : tier;
}

The cascade

Next comes the cascade. Start at the routed tier, call the model, validate. If it passes, return it. If it fails, move up one tier. And if even the large model fails, don't return its answer: hand it to a person. Validation is what makes cheap models safe: a schema with Zod, plus checks for each task (the label is in the list, the invoice lines add up to the total). Every attempt is also logged with its tier and its cost. You will need that log in the next section.

// llm/ask.ts
export async function ask(req: LlmRequest): Promise<ModelAnswer> {
  for (let tier = route(req); ; tier = next(tier)) {
    const answer = await callModel(MODEL_ID[tier], req);
    const check = validate(req, answer);
    log({ task: req.task, tier, ok: check.ok, cost: answer.cost });

    if (check.ok) return answer;
    if (tier === 'large') throw new NeedsHuman(req, check.reason);
  }
}

const validate = (req: LlmRequest, answer: ModelAnswer) =>
  req.schema.safeParse(answer.json).success
    ? TASK_CHECKS[req.task](req, answer) // label in list, lines add up…
    : { ok: false, reason: 'schema' };

Where this breaks

• Silent quality loss: re-run a daily sample on the large model
• Escalation storms: a task that escalates often belongs a tier up
• LLM routers: a model that picks the model adds a call to every request
• Managed routing: one model family, English-optimized, blind to your checks

Four ways this still breaks.

One: silent quality loss. Validation catches broken answers, not mediocre ones. Every day, re-run a small sample of cheap answers on the large model, and compare. That's what the log is for.

Two: escalation. If half the requests for a task escalate, you're paying twice. That task belongs one tier up.

Three: smart routers. A model that chooses the model adds a call to every request. Start with rules: most traffic is predictable.

Four: managed routing. Bedrock's prompt router picks between models of one family, is tuned for English, and can't see your validation results. Fine to start with. Keep your own log anyway.

Targets to hold

• 70%+ of requests on the small tier
• < 5% escalation rate per task
• 0 risky prompts on small

So what does the business get? Most requests answered by the cheapest tier: that's where the savings are. An escalation rate you can count and tune. And a hard rule that no risky question ever lands on the cheap model. The bill goes down, and you can prove quality didn't.

The numbers are targets for an illustrative scenario, not measurements.

▶️ Watch the video: https://www.youtube.com/watch?v=7kZ9mOlxuZo

💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v5-llm-routing

🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1