My Secret Scan Was Green. The Secret Was One Commit Back.

Compliant by Design — Essay 07

Everyone building a pipeline knows to ask whether a security control runs. Almost nobody asks the second question: what, exactly, does it look at? I found the gap between those two questions by planting a fabricated credential in my own repo and watching the secret scanner pass it, green, with the planted key still sitting one commit back from the tip it had just declared clean.

This is the third piece in the factory arc, and it is the mechanism the last one only named. Essay 06 said a green scan is a claim about what was scanned, not about what exists. Here is how a scanner earns a green it has not earned.

The upgrade I would not take on trust

The trigger was boring. GitHub is removing the node20 runtime from its hosted runners on 16 September 2026, and the version of gitleaks-action I was running, v2, depends on it. After that date v2 simply stops working. So the move to v3 was forced, not chosen.

The vendor’s note described it as a runtime bump only. That may well be true. But a secret scanner is the control standing between a careless commit and a public credential, and “trust me, nothing changed” is exactly the sentence you cannot accept about the one thing whose whole job is to not be trusted. So I did not merge it on the changelog’s word. I commissioned it the way I commission any control: by trying to get something past it.

How I commissioned it

The method is the same one I used for the pre-commit hook. Plant a fake secret, watch CI go red, remove it, watch CI go green. A control you have only ever seen pass is a control you have not tested.

One detail mattered more than it looks. I created the bait commit server-side, through the GitHub API, never on my laptop. That meant the local pre-commit hook never touched it: no bypass flag, no quietly unsetting the hooks path, nothing to explain away. It is also the more honest test, because the CI secret scan exists precisely to catch the commits that did not pass a local hook. Web-UI edits, a clone someone never configured, a teammate who skipped the hook on purpose. Testing the CI scanner with a commit the hook had already blessed would prove nothing at all.

The bait that did not fire

Before planting anything in CI, I ran the bait against gitleaks locally to make sure it was detectable. My first candidate was an AWS-format key. Gitleaks did not flag it. Only when I switched to a Stripe-format credential did the local scan light up.

Sit with what nearly happened. If I had planted the AWS string, CI would have stayed green, and I would have concluded the upgraded scanner was broken and gone off to debug a control that was working fine. The probe needs its own positive control. A test that cannot prove it would have failed proves nothing when it passes.

Green, with the secret one commit back

Here is the part I did not expect. I planted the Stripe-format credential, CI went red, good. I removed it in the next commit and pushed. The removal run went green, with the commit that still contained the credential sitting as the direct parent of the new tip.

gitleaks-action decides its scan scope from the event that triggered it. On a push it scans only the commits in that push. On a pull_request it scans the pull request’s full commit list. I confirmed it both ways: a clean tip whose parent held the secret passed the push scan and failed the pull-request scan. The same repository, the same commit graph, two different answers depending on which event did the asking.

And the workflow carried a comment, written in my own hand, asserting that gitleaks scans history and not just the tip. It was false for pushes. It was also sitting in a job that never runs gitleaks at all. I fixed the comment and the scope assumption behind it in their own pull request.

Why I did not file it as “works fine”

The honest reading is that this is not a hole. My main branch is protected. Nothing lands except through a pull request, and the pull-request scan does cover the full history. The push-event scan being narrow does not let a real secret reach main. It would have been very easy to write “works as intended” and close the tab.

I did not, because the finding is not about my branch protection. It is about the shape of the mistake. Everyone tests whether a control runs. The failure that gets you is a control that runs perfectly and looks at the wrong thing. It passes every check you think to run, because you built those checks around the scope you assumed it had, and it fails the single case you installed it for. Scope stays invisible until you force it to show itself.

What a regulator actually asks

This is where it stops being a git story. Under the Notifiable Data Breaches scheme, the question after a suspected exposure is not “was your scanner green”. It is “what personal information was actually reachable, and can you evidence it”. Those are different questions, and “our secret scan passed” answers the wrong one if the scan’s scope never covered the commits in play.

I build for Australians over 50 who have already been scammed once. The assurance I can give has to match what was actually examined, not what a green check implies. Scope of assurance is the whole game, and a green badge that quietly means “the three commits in the last push” is not the same evidence as one that means “everything reachable”.

The uncomfortable takeaway

I only caught this because I refused to trust a bump the vendor called trivial, and because I tested the test before I trusted its result. Both felt like overkill at the time. The version of me in a hurry merges the v3 bump on the changelog’s word, never plants the bait, and ships a repo whose secret scanner is green for a reason he could not state if asked. Green is not an answer. “Green, and here is exactly what was scanned” is. If you cannot finish that second sentence about your own pipeline, you do not yet know what your controls are telling you.

Compliant by Design is a fortnightly series on building AI products under Australian regulation — Privacy Act, APPs, NDB, health data — written from inside the work, not above it.