A conventional web application tells you when it breaks. A request 500s, an exception lands in your error tracker, someone gets paged. The feedback loop between failure and awareness is measured in seconds.
LLM applications do not work like this. They almost never crash. They return a confident, well-formatted, plausible answer that happens to be wrong, and the request completes with a 200. Nothing in your monitoring stack has an opinion about it. The failure is discovered weeks later by a customer, or by a colleague who happened to know the correct answer and noticed the system did not.
This is the single most important operational difference between AI features and the software around them, and it is why so many teams find themselves with something that demoed beautifully in March and is quietly untrusted by September. What follows is a catalogue of how these systems actually degrade, and what you have to instrument to see it happening.
Failure mode 1: Confident wrongness with no error signal
The model does not have a concept of "I could not complete this request." Given an ambiguous input, a missing document, or a question outside its retrieved context, it produces its best continuation. That output is syntactically valid, tonally appropriate, and frequently incorrect.
In a support assistant this shows up as an invented refund policy. In a data extraction pipeline it shows up as a plausible number in a field where the source document had no number at all. In an internal knowledge tool it shows up as a policy the company retired two years ago, stated in the present tense.
How to detect it. You need a ground truth channel, because the system will not volunteer one:
- Structured outputs with schema validation. If the model must return typed JSON, a malformed or out-of-range value becomes a real, catchable error rather than prose. This converts a category of silent failures into loud ones, which is the entire point.
- A required citation for every factual claim. In retrieval systems, an answer without a resolvable source identifier is a defect. Log the rate. If eight percent of answers cite nothing, you have an eight percent problem you were not measuring.
- An offline evaluation set. Fifty to two hundred real inputs with known-correct outputs, run on every prompt or model change. Not a benchmark — your inputs, your edge cases, the three questions that embarrassed you in the pilot.
- Explicit abstention. The model must be able to say it does not know, and you must measure how often it does. A system with a zero percent abstention rate on a corpus that does not cover every question is not confident. It is guessing.
Failure mode 2: Prompt drift after a model update
Your prompt was tuned against a specific model version. The provider ships an update. Nothing in your code changed, your tests still pass, and the output distribution moved underneath you.
Sometimes it moves in your favour. Often it does not: a model that now refuses a category of request it previously handled, formatting that shifts just enough to break a downstream parser, verbosity that quietly doubles your token bill, or a reasoning change that alters classification boundaries in a scoring pipeline.
How to detect it. Pin model versions explicitly and treat an upgrade as a deployment, not a background event. Run your evaluation set against the new version before switching. Track output length distribution over time — a step change in mean tokens per response is one of the earliest and cheapest signals that something moved.
Failure mode 3: Retrieval that degrades as the corpus grows
This one is nearly universal and it is worth understanding precisely, because the system gets worse exactly as it becomes more useful.
At launch, your knowledge base has four hundred documents. Retrieval returns the right chunks because there is little to confuse it. Eighteen months later there are twelve thousand documents, including three superseded versions of the same policy, a draft nobody deleted, and a slide deck that paraphrases the policy inaccurately. Vector similarity does not know which of those is authoritative. It knows which is semantically closest to the question, and the draft is often closer, because drafts are written in the language people actually use.
The retrieval quality curve bends downward with corpus growth, and nothing in a standard dashboard shows it. Your latency is fine. Your error rate is zero.
How to detect it. Log the retrieved chunk identifiers for every query, not just the final answer. Sample and review weekly. Watch for: the same document appearing in the top results for unrelated questions (a chunking problem), superseded documents appearing at all (a corpus hygiene problem), and a rising share of answers where the cited chunk does not actually contain the claim (a re-ranking problem). We wrote a full procedure for this in how to audit a RAG pipeline that is returning the wrong answers.
Failure mode 4: Cost drift
Token spend does not spike. It creeps. A prompt gains a few examples during tuning. Conversation history accumulates because nobody set a window. A retrieval step returns eight chunks instead of four because someone widened it to fix a recall complaint. An agent loop that usually terminates in three steps occasionally runs to twelve.
Each change is defensible. Together they multiply, and because the bill arrives monthly and the growth is roughly linear with usage, it reads as success until someone divides cost by active user.
How to detect it. Instrument cost per operation, not cost per month. Every request should log its input tokens, output tokens, model, and the business operation it served. Then chart cost per completed task over time. A rising line there is unambiguous — it is not more users, it is more expensive users. Set hard token ceilings per operation and alert on the ceiling being hit, because that is a design failure surfacing.
Failure mode 5: Latency creep past the point of usefulness
An answer that arrives in two seconds is a product. The same answer in fourteen seconds is a form people abandon. Latency in LLM systems accumulates across stages that each seem reasonable: embedding the query, retrieving, re-ranking, a first model call to decompose the question, a second to answer, a validation pass, a formatting pass.
The individual budgets are all defensible. The sum is not, and it grows every time someone adds a step to fix a quality problem.
How to detect it. Trace every stage with per-span timing, and monitor p95 rather than the mean — the mean hides exactly the tail that drives abandonment. Set a latency budget for the whole operation before building, and treat a new pipeline stage as spending from it. Streaming the first tokens helps perceived latency and is worth doing, but it is a mitigation, not a fix.
Failure mode 6: Silent tool failures inside agent loops
An agent calls a tool. The tool returns a 429, or an empty result, or a timeout. A well-built agent handles that explicitly. A typical agent treats the failure as an observation, reasons about it, and produces a final answer anyway — one that omits whatever that tool was supposed to contribute, with no indication anything is missing.
The output looks complete. It is missing the data the user actually needed.
How to detect it. Log every tool invocation with its parameters, latency, and outcome. Alert on tool error rates independently of the overall request success rate, because those two numbers can diverge completely. Where a tool result is essential to a correct answer, the agent must fail loudly rather than degrade gracefully — graceful degradation is the wrong default when the user cannot see what was dropped.
The observability that AI systems actually need
Standard application monitoring answers "did it respond, how fast, and did it throw." For LLM systems you also need to answer:
- What did it retrieve? Chunk identifiers, scores, and the final assembled context.
- What did it decide? For agents: every step, tool call, parameter, and result.
- What did it cost? Tokens in and out per request, tied to a business operation.
- Was it right? Sampled human review, user feedback signals, and a scheduled evaluation run.
- Did it decline? Abstention and refusal rates, tracked as first-class metrics.
Points one through three are engineering work you do once. Point four is an ongoing process and the one most teams skip, which is why they discover quality problems from customers instead of from dashboards.
Where this leaves you
If you have an AI feature in production and cannot currently answer "what percentage of answers last week were wrong," that is not a monitoring gap you will get to next quarter. It is the whole risk profile of the feature.
The fix is rarely a better model. It is structured outputs so failures become errors, citations so claims are checkable, evaluation sets so changes are measurable, cost and latency instrumented per operation, and a sampled human review loop that runs whether or not anyone complains.
That is the work an AI audit and rescue engagement does first, before touching prompts — because until the system can tell you how it is failing, every change you make to it is a guess.
If you are running an LLM feature you have stopped fully trusting, tell us what it is doing. We will give you a direct read on whether it is a prompt problem, a retrieval problem, or an architecture problem, which are three very different repair bills.