I audited my own AI product against the Privacy Act. The law probably exempts me. I did it anyway.

Compliant by Design — Essay 04

I run a small software company. One live product: VeriShield, an AI scam checker for Australians over 50. Users paste a suspicious text or email into it, and an LLM tells them whether it looks like a scam.

Here is an uncomfortable fact about businesses like mine: the Privacy Act 1988 almost certainly does not apply to us. The small business exemption excludes most businesses with annual turnover under $3 million, and mine is a bootstrapped solo company a long way under that line. We don’t provide a health service and we don’t trade in personal information, so the standard carve-outs that would pull a small business back into the Act don’t bite either.

So legally, I could stop thinking about the Australian Privacy Principles right now.

I audited the product against all thirteen of them anyway. This essay is about why, how, and what I found, including the findings that stung.

Why audit yourself against a law that exempts you

Four reasons, in descending order of hard-headedness.

First, the exemption is living on borrowed time. Removal of the small business exemption is squarely on the agenda for the next tranche of Privacy Act reform: the government supports it in principle, though as of this writing no Bill has been introduced and no commencement date exists. And the exemption is already being carved away in practice: since 1 July 2026 it no longer applies to small businesses providing AML/CTF designated services, which pulled real estate agents, lawyers, accountants and conveyancers into the Act overnight. VeriShield is not a designated service, so nothing changed for us. But building a product on the assumption that the exemption will keep protecting you is building on a foundation the government has flagged for demolition, and has started dismantling at the edges.

Second, the exemption doesn’t travel. VeriShield has UK expansion on its roadmap, and the UK GDPR has no small business exemption. The cheapest time to build APP-shaped data practices is before the data exists, not during an expansion sprint.

Third, your privacy policy binds you even when the Act doesn’t. If a published policy overstates what you actually do, that is misleading conduct under the Australian Consumer Law, which applies to every business at every size. A privacy policy is a promise, and the ACCC does not care about your turnover.

Fourth, and honestly the one that moves me most: my users are over 50 and my product exists because people are trying to defraud them. Trust is not a feature of this product; it is the product. A privacy failure would kill it regardless of what any regulator did.

How the audit worked

The method is not exotic. It is the method I would use on any client’s product, pointed at my own.

Step one: a data inventory. One row per category of personal information the product touches. Six columns: what it is, where it lives (service and region), why it’s collected, who can access it, how long it’s kept, how it gets deleted. Accounts, payments, the scam-check submissions themselves, LLM API traffic, family-connection records, email, logs, support mail.

Step two: verify against reality, not memory. The first pass was built from documentation. It came back with a scatter of UNKNOWNs: cells where no document could answer the question. The second pass closed them with a read-only sweep of the actual codebase and the actual dashboards: the database console, the email provider, the LLM provider’s account settings.

That distinction matters more than any individual finding. Every UNKNOWN in a data inventory is itself a finding. If you cannot say what region your database is in without logging in to look, then you did not know where your users’ data was. I didn’t. Which brings us to the findings.

What I found

Nearly everything is in the United States

My primary database, hosted on Railway, is in California. I learned this during the audit, not before it. The LLM API is Anthropic, US-based. So are Stripe for payments, Resend for email delivery, and RevenueCat for subscription management. Of every service in the stack, exactly one, the Zoho support mailbox, turned out to be hosted in Australia.

None of this is a scandal. It is the default shape of a modern web stack, assembled from the best-in-class tool at each layer. But under APP 8, sending personal information overseas is a cross-border disclosure, and the discloser stays accountable for what the recipient does with it. When a user pastes a scam text into my checker, that message, which routinely contains their name, their phone number, sometimes fragments of their bank details, crosses the Pacific to be analysed.

The obligation that creates is not “host everything in Sydney.” It is: know exactly which recipients are overseas, understand their retention terms, and say so plainly in your privacy policy. My policy is now being updated to name every offshore recipient, as part of a legal review that was already scheduled.

On the LLM specifically, the retention terms turned out to be better than I had assumed but I had never actually verified them: under Anthropic’s standard commercial API terms, inputs and outputs are deleted within about 30 days and are not used to train models without express permission. I now have dated PDF copies of those terms on file. If you are building on any LLM API and you have not read the retention page of your specific agreement tier, that is a fifteen-minute task that belongs at the top of your list.

The same sweep caught the product’s one remaining third-party connection on the frontend: a Google Fonts stylesheet import that sent every visitor’s IP address to Google on page load. The fonts are now self-hosted and the import is gone.

My only backup was one manual snapshot

This is the finding that actually frightened me. The database plan I was on does not include automated backups. The sole recovery point for my production database, accounts, subscriptions, the family-protection graph, all of it, was a single manual snapshot I had taken two weeks earlier, before a security patch.

A volume failure on the wrong afternoon would have lost real user data with no recovery path. That is an APP 11 problem in the technical sense, but calling it a compliance finding undersells it. It is an existential product risk that a compliance lens happened to catch.

The fix was undramatic: a scheduled nightly database dump to off-platform storage as interim cover, an actual restore test to prove the dumps are usable, and a scheduled upgrade to a plan with proper backups before the product takes live payments. Total effort: an evening.

The AI had $3.68 of runway

While verifying the LLM account settings, I found the API credit balance sitting at $3.68 with auto-reload switched off. If those credits had run out, the product would not have gone down. It would have done something quieter and worse: silently degraded to its local rule-engine fallback, and every check would have returned a weaker verdict with no error, no alert, and no visible difference.

I know how invisible that failure mode is because a version of it had already happened once, for five weeks (15 June to 20 July), when a model was retired out from under an earlier integration. Auto-reload is now on, and credit exhaustion has joined the monitoring backlog as a first-class alert condition.

The lesson generalises to every AI product with a fallback path: graceful degradation without loud degradation is a trap. The better your fallback, the longer you won’t notice you’re running on it.

No email we sent identified who we are

Every outbound email template, the magic links, the password resets, the family invitations, the danger alerts, was checked for sender identification. The company’s legal name appeared in none of them. Zero of five.

The Spam Act’s sender-identification rules attach to commercial electronic messages, and whether a transactional welcome email that mentions a paid tier is “commercial” is a genuinely arguable legal question. I am not going to argue it, because adding the entity name, ABN and contact address to five email footers costs nothing and moots the question entirely. Fixed.

VeriShield has a family-protection model: an adult, often a grown child, sets up protection that involves another adult, often a parent. That structure raises a consent question the Privacy Principles were not really drafted around: whose information is being collected, and who agreed?

The current design does the things I would tell a client to do. The protected person is not silently enrolled: they must create an account, sign in, and pass a consent screen that names the person who invited them before any connection activates. Until they consent, the product holds nothing about them except the email address the invitation was sent to.

The audit still surfaced the sharpest disclosure question in the product, and it is one no checklist would have found: accepted contacts can see the full text of the account owner’s ten most recent scam checks. That is the feature working as designed, it is what family protection means, but it is also the single least obvious thing a user is agreeing to. The consent copy is being revised to say it in plain words, as part of scheduled legal review. If your product has one fact users would be surprised to learn, your consent flow should state that fact more prominently than anything else.

The plan I should have written on day one

The audit asked a simple question: if there were a breach tomorrow, is there a written plan? There was not. Under the Notifiable Data Breaches scheme, an eligible breach starts a clock: contain, assess whether serious harm is likely (the formal assessment has a 30-day outer limit, but for a product this size it should take days), and if it is, notify the OAIC and the affected people as soon as practicable.

None of that is hard. All of it is much harder to improvise mid-incident. There is now a one-page plan: who assesses (me, there is no one else), what the clock is, how users get contacted, where the OAIC form lives, and, because the audit proved detection was my weak point, a standing list of the monitoring gaps that would stop me noticing a breach in the first place.

I have published the template alongside this essay at ugoharry.com/breach-response-plan. Take it, replace the names, and you have closed the same gap in an afternoon. The review lens I used is public as well, as two Claude Code skills: github.com/UgoHarry/au-compliance-pack.

What I’m not telling you

A handful of findings from the full register are not in this essay, because they are not fixed yet. That is a deliberate rule, and it is one I would give any founder writing about their own security or privacy posture: audit in full, publish what is fixed or safe, and never narrate a live weakness. Transparency about your process does not require handing out a map of your open gaps. The remaining items are scheduled, and the lawyer reviewing the privacy policy gets the unabridged register.

What this cost, and what it bought

The inventory and verification took an evening plus a fifteen-minute sweep of dashboards. The immediate fixes took another evening. The expensive-looking items, the policy rewrite and consent-copy review, were already on the plan; the audit just gave the lawyer a complete and honest input instead of a hopeful one.

For that price: I now know where every category of user data lives, in which country, under which retention terms, with a tested backup, a breach plan, and a written register of what remains. Six weeks ago I could not have answered the most basic question a regulator, an acquirer, or a worried user could ask: where is my data?

If you are building an AI product in Australia and the small business exemption is your privacy strategy, I’d gently suggest it isn’t one. The audit that replaces it costs two evenings.

This essay describes a practitioner’s gap analysis of my own product. It is not legal advice, and the legal positions described (the small business exemption analysis in particular) are flagged for review by an actual lawyer, which is exactly where your edge cases should go too. Framework references: Privacy Act 1988 (Cth), Australian Privacy Principles, Notifiable Data Breaches scheme, Spam Act 2003, verified against OAIC guidance at the time of writing.

Compliant by Design is a fortnightly series on building AI products under Australian regulation — Privacy Act, APPs, NDB, health data — written from inside the work, not above it.