The AI That Invented a Source
Compliant by Design — Essay 05
If you let an AI draft anything the public will read, this is the failure your tests will not catch. Mine got as far as a rendered homepage, a clean build and a passing verification table before a human asked one question.
What happened
On 24 August I had Claude Code rebuild the logged-out homepage for VeriShield, a scam-protection product for Australians over 50. The design had a band of statistics near the top, and under the band sat a slot for a source line, the small grey text that says where the numbers come from.
Two of the numbers were old. “84% of over-50s have encountered a scam.” “43% can’t tell an AI voice from a real one.” They had been in the app copy for months. I did not know where they came from. Neither did the model.
The layout needed a source line. The model wrote one: “Awareness figures: Australian consumer scam research, 2025.”
No such research was ever identified. Not misattributed, not loosely paraphrased. Invented, to fit the slot.
Why every check passed
I run a verification table on every AI pass: the build passes, a grep for banned strings, accessibility checks, screenshots. All green. The page looked authoritative in the screenshot precisely because the source line was there. A citation is what authority looks like.
None of those checks could have caught it. The failure was not in the code. The HTML was valid, the text was well formed, the accessibility tree was fine. The failure was in whether a sentence was true, and I have no linter for that.
It was caught in review, by asking one question of every number on the page: where does this actually come from? Two of the three had no answer. The third had one.
The mechanism
The model did not lie unprompted. I want to be precise about this, because the easy version of the story, “AI hallucinates, be careful,” teaches nothing.
The design had a slot that demanded a citation. The numbers had none. The model resolved the contradiction the way a rushed junior copywriter would: it produced text of the right shape. Structure demanded a source, so a source appeared. I built the pressure; the model relieved it.
That reframing matters because it tells you where the fix goes. Not “prompt the model to be honest.” Remove the slot, or fill it before the model gets there.
What I did
I cut both unsourced numbers from the marketing page rather than re-source them under time pressure. A statistic you cannot trace is not a statistic, it is a claim in a costume. The band was rebuilt around the one figure I could trace to a primary source: $2.18 billion in reported losses in 2025, from the National Anti-Scam Centre’s Targeting Scams report, published 30 March 2026.
The same two numbers still sit in the signed-in app copy. They are on the fix list before launch, filed as a content defect in the same register as a broken button.
The regulated angle
This product’s entire pitch is “can you trust what you are reading?” It nearly answered that question on its own homepage.
Two things make that more than embarrassing. The Australian Consumer Law prohibits misleading or deceptive conduct in trade, and a fabricated citation under a statistic is a representation about the evidence behind a claim. I am not a lawyer, and I did not want to find out where that line sits. And the audience is people who have already been burned once by confident, official-looking text. Publishing more of it, even by accident, is the one thing this product must never do.
The compliance lens I already apply to code, with its “reasonable steps” and “documented” and “verified,” turned out to have a hole in it. It covered what the model built and not what the model wrote.
The rule now
Every number in AI-drafted public copy gets a provenance check before it ships. No source, no number. The check is a table: figure, source URL, what the source actually says, date verified. If a row cannot be filled, the figure is cut, not softened.
Two weeks later I ran that same table over a repository before making it public. Twenty-six regulatory claims. None fabricated, three stale. One had gone stale six days earlier, when an exposure draft of a Bill was released. The gate paid for itself on its first run.
The uncomfortable takeaway
I had automated every check I knew how to automate, and the thing that got through was the one that looked most like a finished product. A citation is the cheapest possible way to make text look verified. If you are shipping AI-written words to the public and your review stops at “does it build,” you are not reviewing the part that can hurt someone.
Compliant by Design is a fortnightly series on building AI products under Australian regulation — Privacy Act, APPs, NDB, health data — written from inside the work, not above it.