AI Product Debt: Evals, RAG, and Agents

What is AI Product Debt?

Eval Debt: The One That Hides All the Others

"It looked good in the demo" is the LLM equivalent of "it works on my machine." Both are statements about a single observation on a hand-picked input, offered as evidence about a population. The difference is that "works on my machine" is a punchline the industry learned to distrust decades ago, and "looked good in the demo" is still, in a lot of organisations, how AI features get approved for production.

What an eval suite usually looks like when it has debt

// evals/run.js - written in sprint 1, last edited 14 months ago
const CASES = [
    { input: "summarise the Q3 report", note: "should mention revenue" },
    { input: "summarise the complaint", note: "should mention the refund" },
    { input: "summarise the contract",  note: "should mention the term" },
];

for (const c of CASES) {
    const out = await summarise(c.input);
    console.log("---", c.input);
    console.log(out);          // a human reads the console and nods
}

Golden datasets go stale

A golden dataset is a snapshot of what your users asked for at the moment you wrote it. Users change. Your product changes. The dataset does not. Six months later your eval suite is measuring performance on a distribution of questions that no longer resembles production traffic, and it will report a healthy score while real users hit a category of request that did not exist when the dataset was frozen.

Worse, golden datasets are usually built by the same person who wrote the prompt, from the cases they were already thinking about. That is a closed loop. It measures the intersection of what the author imagined and what the author already handled, which is exactly the region where the system is guaranteed to do well. Treat a golden dataset as a living artifact with an owner and a refresh cadence, sampled from real traffic, not as a fixture checked in once.

{
  "dataset": "summarisation",
  "version": 7,
  "owner": "search-team",
  "refreshed": "2026-07-01",
  "refresh_cadence_days": 90,
  "sampling": "stratified random sample of production traffic, plus every case from an incident",
  "cases": 412,
  "held_out": 84
}

Eval sets leak into prompts

This is the AI-product version of training on your test set, and it happens by accident almost every time. An engineer finds a case the model gets wrong, and fixes it the fastest way available: by pasting that exact case into the prompt as a few-shot example. The eval score goes up. Nothing generalised. The suite is now scoring the model on inputs it has been handed the answers to.

The fix is mechanical, and it belongs in CI rather than in a code review checklist, because a reviewer will never notice a fifteen-word overlap between two files they are not reading side by side.

// ci/check-eval-leakage.js
// No eval input may appear verbatim inside any shipped prompt template.
const prompts = loadAllPromptTemplates();     // every .md / .txt under prompts/
const dataset = loadDataset("summarisation");

const leaks = [];
for (const c of dataset.cases) {
    const needle = normalise(c.input).slice(0, 60);
    for (const p of prompts) {
        if (normalise(p.body).includes(needle)) {
            leaks.push({ case: c.id, prompt: p.path });
        }
    }
}

if (leaks.length) {
    throw new Error(
        `Eval leakage: ${leaks.length} eval case(s) are embedded in prompts.\n` +
        leaks.map(l => `  ${l.case} -> ${l.prompt}`).join("\n")
    );
}

Keep a held-out slice that no engineer is allowed to look at while iterating on prompts. If the held-out score and the working score diverge, you are overfitting your prompt to your eval set, and the gap between them is the size of the lie your dashboard is telling you.

Measuring only the cases you already fixed

Most eval sets grow by accretion from bug reports. Something breaks, someone adds a case, someone fixes the prompt. Repeat for a year and you have a suite composed entirely of solved problems. Its score approaches 100% and stays there, and it has essentially zero power to detect anything new. A suite that never fails is not a passing suite, it is an uninformative one. Deliberately keep a proportion of cases that the system currently fails, and track that proportion; when it hits zero, your suite has stopped doing work.

Regression testing for a non-deterministic system

The reason teams skip regression testing here is that the naive approach does not work: you cannot assert on an exact string when the same input produces different output on consecutive calls. The answer is to stop asserting on outputs and start asserting on a scored distribution. Run each case several times, score each run with a mix of deterministic and model-graded checks, and compare the aggregate against a stored baseline with an explicit tolerance.

// evals/summarisation.eval.js
const results = await runEval({
    dataset: loadDataset("summarisation", { version: 7 }),
    variant: process.env.PROMPT_VARIANT ?? "current",
    repeats: 5,                       // sample the distribution, do not sample once
    scorers: [
        requiredSections,             // deterministic: all four headings present?
        lengthBudget,                 // deterministic: under 200 words?
        validJson,                    // deterministic: parses against the schema?
        groundedness,                 // model-graded: every claim traceable to source?
        refusalRate,                  // did it decline a request it should have handled?
    ],
});

assertNoRegression(results, {
    baseline: "main",
    tolerance: {
        groundedness:     -0.02,      // 2 points of noise is acceptable
        requiredSections:  0.00,      // deterministic checks get zero tolerance
        validJson:         0.00,
    },
});

For the broader testing picture, including why coverage numbers mislead in AI-adjacent codebases, see AI testing gaps.

RAG Debt: Silent Drift in the Retrieval Layer

Retrieval-augmented generation moves the hard problem out of the model and into a pipeline nobody owns. The model is fine. The prompt is fine. The answer is wrong because the chunk it needed was never retrieved, or was retrieved from a version of the document that was superseded in March.

Index staleness

An index built by a one-off script during the prototype phase has no owner, no schedule, and no alarm. Documents get edited, deleted, and superseded in the source system while the vectors sit unchanged. The failure is invisible: the system confidently answers from the old text, and it looks exactly like a correct answer. Deleted documents are the sharpest edge here, because a deleted source document that still has live vectors turns your assistant into a machine for resurfacing information somebody deliberately removed.

Treat freshness as a service level objective with a number attached, and emit it as a metric rather than trusting a cron job to have run.

// Freshness is a measurable SLO, not a hope
const STALENESS_BUDGET_HOURS = 24;

const oldest = await vectors.aggregate({ min: "metadata.indexed_at" });
const ageHours = (Date.now() - Date.parse(oldest)) / 3_600_000;

metrics.gauge("rag.index.oldest_chunk_age_hours", ageHours);
metrics.gauge("rag.index.orphaned_chunks", await countChunksWithNoLiveSource());

if (ageHours > STALENESS_BUDGET_HOURS) {
    alert.page("RAG index is stale", { ageHours, budget: STALENESS_BUDGET_HOURS });
}

Chunking chosen once, never revisited

Every RAG system has a line like this, copied from a tutorial in the first week, and it is load-bearing forever.

# Chosen on day three. Never measured. Never changed.
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200

splitter = RecursiveCharacterTextSplitter(
    chunk_size=CHUNK_SIZE,
    chunk_overlap=CHUNK_OVERLAP,
)

Those two numbers determine whether a table survives intact, whether a policy clause keeps the sentence that qualifies it, and whether a code block is split down the middle. They were tuned against the documents you had on day three. The corpus you have now may be mostly spreadsheets and meeting transcripts, for which a fixed character window is close to the worst possible strategy. Chunking parameters are configuration that deserves an eval, a changelog entry, and a re-run whenever the shape of the corpus changes materially.

The practical move is to record the chunking strategy in the chunk metadata so that a mixed index is at least detectable, and so a future re-index can be done incrementally rather than as one terrifying big-bang job.

Retrieval quality degrades as the corpus grows

A retrieval setup tuned on two thousand documents behaves differently at two hundred thousand. More near-duplicates compete for the same top slots. A similarity threshold that used to exclude junk now excludes relevant material, or stops excluding anything at all. A fixed top-k that was generous becomes a bottleneck. None of this raises an error; it just makes answers slightly worse, slowly, in a way that no individual user reports because each one only sees their own query. This is the strongest argument for retrieval-specific evals that are separate from end-to-end answer evals: you want to know that recall dropped before you find out that satisfaction did.

No observability into what was retrieved

The most common and most expensive RAG debt is simply not logging the retrieval step. When a user reports a wrong answer, the only question that matters is whether the right chunk was retrieved and ignored, or never retrieved at all. Those two failures have completely different fixes, and without a retrieval log you cannot tell them apart, so every investigation degenerates into prompt guesswork.

log.info("rag.retrieval", {
    trace_id,
    query_hash: hash(query),          // hash, not the raw query, if it is sensitive
    strategy: "hybrid/bm25+dense",
    top_k: 8,
    threshold: 0.32,
    hits: hits.map(h => ({
        chunk_id: h.id,
        source_doc: h.metadata.source_doc,
        source_sha: h.metadata.source_sha,
        score: h.score,
        rank: h.rank,
        indexed_at: h.metadata.indexed_at,
    })),
    cited_in_answer: citedChunkIds,   // which ones the model actually used
    latency_ms: elapsed,
});

The cited_in_answer field is the one people forget, and it is the one that pays. The ratio of retrieved chunks to cited chunks tells you whether your top-k is too wide, and a query where nothing retrieved was cited is a strong signal that the model answered from its own parameters rather than from your documents. See observability debt for the general version of this argument.

Vector Store Migration Debt

This one deserves its own section because it is the AI product equivalent of a database schema migration, and it is routinely underestimated by an order of magnitude. Embeddings from one model are not comparable to embeddings from another. There is no conversion function. When you change the embedding model, every vector you have ever stored becomes garbage simultaneously, and the only remedy is to re-embed the entire corpus.

Here is what that migration actually involves, which is why it never fits in the sprint it gets planned into.

Changing the embedding model: the real checklist

1. Dimensionality usually differs, so this is a NEW index, not an ALTER.
   Capacity planning happens twice: you run both indexes for a while.
2. Re-embed every chunk in the corpus. Cost scales with total tokens,
   not document count, and long-tail documents dominate that total.
3. Dual-write during the transition so nothing indexed mid-migration is lost.
4. Dual-read behind a flag so you can compare old and new on live traffic.
5. RE-TUNE top-k and the similarity threshold. Scores from a different model
   are on a different scale. A 0.78 cutoff carried over from the old index
   is a number with no meaning in the new one.
6. Re-run the eval suite to prove retrieval did not regress.
   (This is the step that reveals you do not have an eval suite.)
7. Decide the rollback story BEFORE cutover: keeping the old index warm
   costs money, and deleting it makes rollback a multi-hour re-index.

Step five is where most of the silent damage happens. Teams migrate the vectors, keep the thresholds, and ship. Retrieval quality changes, sometimes for the better and sometimes not, and because nobody re-tuned the cutoff the change is attributed to the model rather than to the configuration.

The deferral itself is the debt. Teams look at that list, correctly conclude it is a multi-week project, and postpone it. Meanwhile the embedding model they are on ages, gets deprecated, and eventually acquires an end-of-life date, at which point the multi-week project becomes a multi-week project with a deadline attached and no slack. The way out is to make the migration cheap before you need it: store enough metadata on every chunk that a re-index is a scripted operation rather than an archaeology exercise.

{
  "chunk_id": "handbook/leave-policy#3",
  "source_doc": "handbook/leave-policy",
  "source_sha": "9f2c1ab4e7d0",
  "source_modified_at": "2026-06-30T11:04:00Z",
  "embedding_model": "provider-embed-gen3",
  "embedding_dim": 1536,
  "chunker": "recursive/1000-200",
  "indexed_at": "2026-07-25T14:02:11Z",
  "pipeline_version": "4.1.0"
}

Agent and Tool-Definition Debt

A tool definition is a prompt. The name, the description, and the parameter descriptions are all read by the model and all influence behaviour. Teams treat them as documentation, review them like documentation, and test them not at all.

Tool descriptions are untested prompt surface

{
  "name": "delete_records",
  "description": "Deletes records. Use when the user wants to clean up.",
  "input_schema": {
    "type": "object",
    "properties": { "filter": { "type": "string" } }
  }
}

Nothing in that definition tells the model the operation is irreversible, that an empty filter is catastrophic, what the tool returns, or what a timeout means. Somebody wrote it in thirty seconds and it has been in production for a year. Compare it to a definition written as if it were what it actually is, which is production behaviour specification.

{
  "name": "delete_records",
  "description": "Permanently delete records matching an explicit filter. DESTRUCTIVE and NOT reversible. You MUST call count_records with the same filter first and confirm the count with the user before calling this. Never call with an empty filter or a wildcard. Returns { deleted: number }. If this call times out the deletion MAY have completed - call count_records to check rather than retrying.",
  "input_schema": {
    "type": "object",
    "required": ["filter", "confirmed_count"],
    "properties": {
      "filter": {
        "type": "string",
        "description": "Non-empty filter expression. Must match the filter passed to count_records."
      },
      "confirmed_count": {
        "type": "integer",
        "description": "The count returned by count_records and confirmed by the user. The call fails if it does not match the current count."
      }
    }
  }
}

Unclear failure semantics and retried destructive operations

An agent that receives a timeout has no way to know whether the operation succeeded. Absent explicit guidance it will do the reasonable-looking thing and try again, which for a non-idempotent operation means doing it twice. This is not a model failure, it is a missing contract. The fix is the same one distributed systems arrived at long ago: idempotency keys, plus a tool result vocabulary that distinguishes "did not happen" from "unknown".

const OUTCOME = {
    OK:        "ok",          // completed; safe to move on
    FAILED:    "failed",      // definitely did not happen; safe to retry
    UNKNOWN:   "unknown",     // may or may not have happened; DO NOT retry
    REJECTED:  "rejected",    // preconditions not met; fix args and retry
};

async function callTool(name, args, ctx) {
    const key = idempotencyKey(ctx.trace_id, name, args);   // stable across retries
    try {
        const result = await registry[name].execute(args, { idempotencyKey: key });
        return { outcome: OUTCOME.OK, result };
    } catch (err) {
        if (err.code === "TIMEOUT" || err.code === "CONNECTION_RESET") {
            // Tell the model the truth instead of letting it guess.
            return {
                outcome: OUTCOME.UNKNOWN,
                guidance: "The operation may have completed. Verify with a read " +
                          "tool before taking any further action. Do not retry.",
            };
        }
        return { outcome: OUTCOME.FAILED, error: err.message };
    }
}

Tools accumulate and are never removed

Tools are added because a feature needed them and kept because removing things is scary. Every one of them consumes context on every single request, and every additional near-duplicate makes selection harder. A registry with a handful of well-described tools outperforms one with dozens of overlapping ones, and the overlapping ones are also where the wrong-tool-selected bugs come from. Treat the registry like a dependency list: it needs an inventory, an owner per entry, and a removal process.

// ci/check-tool-registry.js
for (const tool of registry.all()) {
    assert(tool.owner,        `${tool.name}: no owner`);
    assert(tool.description.length > 80,
                              `${tool.name}: description is too thin to be a prompt`);
    assert(hasIntegrationTest(tool.name),
                              `${tool.name}: no multi-step integration test`);
    assert(tool.lastUsedWithinDays(90),
                              `${tool.name}: unused for 90 days - remove it`);
    if (tool.destructive) {
        assert(/NOT reversible|irreversible/i.test(tool.description),
                              `${tool.name}: destructive tool must say so in prose`);
        assert(tool.requiresConfirmation,
                              `${tool.name}: destructive tool needs a confirmation gate`);
    }
}

No integration tests for multi-step behaviour

Unit-testing each tool proves each tool works. It proves nothing about the loop. The interesting failures are all sequence failures: calling the write before the read, looping between two tools forever, giving up after one failure on a task that needed three steps, or taking a destructive action on a trajectory nobody imagined. Test the trajectory, not just the endpoints.

test("refund flow never deletes before confirming", async () => {
    const trace = await runAgent({
        prompt: "Cancel the duplicate charges on account 4471 and clean them up.",
        tools: sandboxTools(),
        maxSteps: 12,
    });

    const names = trace.toolCalls.map(c => c.name);

    // Ordering constraints, not output assertions.
    expect(names).toContain("count_records");
    expect(names.indexOf("count_records"))
        .toBeLessThan(names.indexOf("delete_records"));

    // No destructive call without a confirmed count.
    for (const call of trace.toolCalls.filter(c => c.name === "delete_records")) {
        expect(call.args.confirmed_count).toBeGreaterThan(0);
    }

    // The loop terminates.
    expect(trace.steps).toBeLessThan(12);
    expect(trace.repeatedIdenticalCalls).toBe(0);
});

Testing a System That Answers Differently Every Time

The objection that stops most teams from testing LLM features at all is real: the output is not stable, so an equality assertion is worthless. The resolution is to stop testing the text and start testing the properties the text must have, and to convert flakiness into a measured pass rate instead of pretending it does not exist.

// Worthless: asserts on a sample of a distribution
expect(answer).toBe("The refund window is 30 days.");

// Useful: properties that must hold for every valid answer
expect(answer).toMatch(/30 days/);                      // the fact is present
expect(await isGrounded(answer, retrieved)).toBe(true); // no unsupported claims
expect(() => JSON.parse(answer)).not.toThrow();         // structural contract
expect(schema.safeParse(JSON.parse(answer)).success).toBe(true);
expect(answer).not.toMatch(/as an AI|I cannot help/i);  // no unexpected refusal

Then run the case many times and assert on the rate, not the run. A single execution of a non-deterministic test tells you almost nothing, and worse, it makes the suite flaky, which trains the team to re-run it until it passes. That habit destroys the value of the suite faster than not having one.

test("extraction returns valid, grounded JSON", async () => {
    const RUNS = 20;
    const results = await Promise.all(
        Array.from({ length: RUNS }, () => extract(FIXTURE_INVOICE))
    );

    const passRate = results.filter(isValidAndGrounded).length / RUNS;

    // The threshold is a product decision, written down, with a reason.
    expect(passRate).toBeGreaterThanOrEqual(0.95);
});

Four practices make this affordable rather than exhausting:

• Separate the deterministic shell from the probabilistic core. Parsing, validation, retries, routing, and tool dispatch are ordinary code and deserve ordinary fast unit tests with the model mocked. Only the model call itself needs the expensive treatment.
• Record and replay. Cache real model responses as fixtures so the majority of CI runs are deterministic and free, and run the live sampled suite on a schedule and before release rather than on every commit.
• Prefer structural contracts. Anything you can force into a schema becomes testable with zero fuzziness. The more of your output that is structured, the less of your test suite depends on judgement.
• Version and eval your model-graded scorers. A grader is itself an LLM feature and can drift for exactly the same reasons. Score the grader against a human-labelled set periodically, or you end up with a thermometer that is quietly wrong.

Prompt-level versioning and change control are close cousins of all of this and are covered in more depth on prompt engineering debt.

A Remediation Roadmap

This sequence assumes you are somewhere around level zero or one and have a product in front of real users. It is ordered by how much each step unlocks the next, not by how hard it is.