# Silent Failure Audit

*A daily checklist for trusting AI (or any) output before you act on it.*

*Author: Bogdan Rynkowski · [BogdanRynkowski.com](https://bogdanrynkowski.com)*

> Companion checklist to the essay *"The AI kept telling me everything was working. It wasn't."* — turning its lessons into something you run, not just read.

## Why this exists

Clean output is not evidence of a correct result. The dangerous mistakes don't crash or throw an error — they produce a plausible-looking number, chart, summary, or answer on a flawed foundation, run after run. This checklist is the layer between *"the output looks fine"* and *"I trust this enough to act on it."*

Run it whenever a result is about to inform a decision — a report you're about to send, a number you're about to quote, a recommendation you're about to act on. Default to suspicion in proportion to how clean and how good the result looks.

- **A result that keeps improving deserves MORE scrutiny, not less.** That's exactly the shape of a metric that looks good enough that you stop looking.
- **A result that confirms what you already expected deserves the same scrutiny.** Agreement is the moment you stop checking — which is exactly the moment a silent error survives.

## The seven checks

Don't skip a check just because the run had no errors. The whole point is that these failures are silent — no crash, no warning, plausible output every time.

### 1. Am I answering the right question?
The headline number can climb while the thing that actually determines success falls.
- State the real goal in one sentence. Is the number you're celebrating *causally* tied to it — or just correlated, convenient, or easy to pull up?
- Identify the number that should move the *opposite* way if you were quietly optimizing a proxy instead of the real goal. Look at it before you celebrate.
- A "correct answer to a subtly wrong question" isn't a mistake by the AI — it's answering exactly what you asked. Re-read what was literally computed versus what you meant.
- **Before running the check, name the result that would change your decision.** If no possible outcome would change what you do next, the check is theater — say so and skip it.
- **Guard against confirmation.** If the result agrees with what you already believed, that's a reason to look *harder*, not a reason to stop.

### 2. Are the outputs based on values that actually exist?
A single gap in the underlying data can make an entire result a function of nothing.
- Check how missing or blank values are handled. Are they dropped, zero-filled, carried forward, or silently ignored?
- Confirm the obvious: does the count of things actually used match the count you expected? Check it; don't assume it.
- Look at the shape of the results. Suspiciously uniform, suspiciously tight, or all-identical usually means a constant is masquerading as an analysis.
- **A blank, zero, or "N/A" is "no signal," not a measured value** — and is often a sign the check never ran at all. Don't report an absent value as if it were a measured one.

### 3. Does a combined result actually contain every piece it claims to?
"A and B combined" routinely turns out to be "B only," because of one detail in how it was built.
- For anything described as a combination of factors, verify each factor actually varies and contributes. A thing labeled "X and Y" can secretly be just Y.
- Check the contribution of each piece. A component that's flat or always the same value is silently absent, no matter what the label says.
- Trust the underlying data over the label. The label is documentation, not proof.

### 4. Did I look at what got rejected or filtered out — not just what passed?
The best candidates can be quietly filtered out and dismissed as "not qualified."
- Pull the rejected, filtered, or excluded items and actually read them. Is anything there that obviously should have made the cut?
- For anything involving dates, time windows, or event boundaries, check the edge cases carefully. An off-by-one error tends to silently cut out the cases closest to the boundary — often the most important ones.
- Before "fixing" something that looks wrong, make sure it isn't intentional or load-bearing. Correcting it without checking can quietly break something that depended on it.
- Trace at least one rejected item all the way through to confirm it was excluded for a real, valid reason.

### 5. Why does this look clean? Name the actual reason.
- Many clean runs aren't proof of correctness — they can all share the same blind spot. A long streak of results with no visible problems is not the same as validation.
- Pick the one result you're most tempted to take on faith, and verify it by hand from the raw underlying data. If you can't, say so plainly instead of implying it's been checked.

### 6. Is the effect bigger than the noise?
Checks 1–5 catch structural mistakes; this one catches statistical ones. Any number that comes from a sample, a simulation, a survey, or a randomized/limited process is an estimate, not a fact — and a precise-looking number can be pure noise.
- **Re-run it (different sample, more data) before trusting its direction or size.** If the result swings by more than the effect you're claiming, you measured noise, not a real signal.
- Sanity-check the method against something you already know the answer to. If a known baseline doesn't come out right, distrust the whole method — not just the new result.
- Treat small-sample or short-window numbers as hypotheses, not conclusions. Name the bigger test that would actually confirm it before acting.

### 7. Are the two things I'm comparing actually comparable?
Every "this is better/worse than before" claim assumes both sides came from the same conditions — same time period, same data, same settings, same sample size. A difference between mismatched conditions is an artifact that looks perfectly clean every time, because nothing about it is technically "wrong" — it's just measuring the wrong comparison.
- Name exactly what should be held constant between the two things being compared, and verify it actually was.
- Watch for stale or partially updated information: if one side reflects more recent data than the other, that gap — not your actual change — is what's driving the "difference."
- When you build something new to compare against an old baseline, make sure it covers the same range/scope. A mismatched scope is a silent artifact.

## Output contract

Whenever you use this checklist, end with a short, honest report — not reassurance:
- **Checked:** which of the seven checks you actually ran, and how (the real number or evidence — not "looks fine").
- **Findings:** anything that was off, with the specific evidence.
- **Not verified:** what you could not confirm, and why. Say it plainly — don't let something unverified pass as verified.
- **Verdict:** trust it or don't, and name the one thing a human should personally double-check before acting on it.

Report problems with the evidence behind them. If you skipped a check, say you skipped it. Never let "no errors" quietly become "correct."