What we considered, why TruffleHog won, how we rolled it out, and the problems we hit doing it.

The customer is an automotive AI company; the engagement is described here with their identifying details removed.

The problem: how do you scan 300+ repositories without spending a fortune?

I joined as an embedded DevOps engineer at a customer whose codebase had grown to roughly 300 GitHub repositories across around 60 organization members, on a GitHub Team plan. Nobody could answer a question that sounds like it should be easy: are there live credentials in our git history?

Three constraints squeezed the answer from different directions. It had to be close to free — preventive hygiene with no incident behind it competes for budget against features, and loses. It couldn’t touch the shared Actions minutes, which on a Team plan are one budget the whole org draws from; anything scanning every push across 300 repositories becomes a tax on everyone else’s pipelines, and the first thing anyone does about that is turn it off. And it needed a centralized rollout, because any per-repository change multiplied by 300 is a project in itself, and it decays the moment repository 301 is created.

What we considered:

• GitHub’s own secret scanning and push protection – the best long-term answer, because it prevents the commit instead of finding it later. But it’s a paid add-on on top of the customer’s plan, and making it mandatory org-wide needs Enterprise-tier required workflows. It became a quote to get, not something I could ship that month.
• Commercial scanning platforms – priced per seat or per repository. Hard to justify for a question we hadn’t answered yet. You don’t buy a platform to find out whether you have a problem.
• Scan-on-every-push, built in-house – cheap per run, ruinous in aggregate. Exactly what the shared-minutes constraint forbids.
• A one-off manual sweep – answers the question once, then answers it wrong a week later.
• An open-source scanner on a schedule – free at the tool level, compute is whatever we choose to spend, and one workflow can iterate over the whole organization instead of 300 repositories each carrying their own.

Only the last category survived all three constraints, so the question narrowed to which scanner. That’s how I got to TruffleHog.

What TruffleHog is, and why it fit

TruffleHog is an open-source secret scanner. Point it at a git repository, a directory, or an entire GitHub organization and it matches against roughly 800 credential detectors. That part is table stakes. Two things made it the right choice here.

It verifies rather than only matching. For detectors that support it, TruffleHog takes the candidate credential and authenticates against the issuing service. A finding marked verified isn’t a string that resembles a key – it’s a key that worked. This changes the economics of the whole project: pattern matching alone produces a report full of maybes, and a report full of maybes is a triage backlog nobody staffs. Verification splits findings into “live, act today” and “could not be tested”, which is the difference between a report that gets worked and one that gets closed.

It scans history, not just the current checkout. A credential’s lifecycle is usually: committed, noticed, deleted from the file, left in git history forever. Reading only the working tree misses all of those.

It’s also Apache-licensed, so no procurement conversation; a single binary, so it runs anywhere Actions runs; and it emits JSON, which meant I could build my own reporting instead of accepting whatever the tool prints.

How we rolled it out

One scheduled workflow, in one repository, covering the whole organization – plus a lightweight gate that catches credentials at pull-request time.

 weekly cron ───────▶ ┌──────────────┐
                      │  preflight   │   resolve reachable repos, fail fast
                      └──────┬───────┘
                             ▼
                      ┌──────────────┐
                      │     scan     │   sharded across parallel jobs
                      │              │
                      │  ┌────────┐  │   git history  ──┐
                      │  │ 2 modes│  │                  ├─▶ raw findings
                      │  └────────┘  │   working tree ──┘   (never leave
                      └──────┬───────┘                       this job)
                             ▼
                 redacted findings only
                             ▼
                      ┌──────────────┐
                      │    notify    │   classify → render → allowlist
                      └──────┬───────┘
                             ▼
                issue, only if critical/high

Access runs through a dedicated GitHub App scoped to the minimum, with its keys as repository secrets rather than organization secrets – an organization secret would expose that private key to the workflows of all 300+ repositories, which is the exact class of problem we were looking for. One blocker worth writing down, because it isn’t in any guide: trufflehog github --token can never work with an App installation token. The connector calls GET /user to resolve a login for git basic-auth, and an installation token has no user behind it. The App path exists but is only reachable through multi-scan --config, so the scan writes that config at runtime under umask 077 and shreds it afterwards.

The three-job split is positioned so only already-redacted output ever crosses a job boundary. Raw findings never leave the scan job, and get shredded on the error path as well as the success path – the error path is where this leaks.

Cost control is one explicit trade: the weekly run covers the working tree plus the last 90 days of commits for large repositories, with a full-history sweep monthly. A credential both older than 90 days and already deleted from the tree is caught monthly rather than weekly. The whole thing runs for a few dollars a month.

Most of the work went into reporting, not detection. A scanner that reports 300 findings hasn’t reduced risk — it’s moved the problem from “we don’t know” to “we don’t have time”. So a composite action sits between TruffleHog’s JSON and anything a human reads, rendering each finding as a few labelled lines: location, masked fields, store in, how (with a filled-in command), rotate. Severity and liveness stay separate axes, so an unverifiable finding reads as “could not be tested; assume live until rotated” and never as safe. Storage advice routes by credential type and file path, because a webhook in a CI workflow becomes an Actions secret while the same webhook in a Helm chart needs a secrets manager. Issues open only for critical or high, which is what lets the signal survive contact with a real engineering team.

Findings and problems we hit

The first full scan came back clean, and that was the bug. Zero findings on the largest repository the customer has. A green scan is what everyone wants to see, so nobody would have questioned it. There were fourteen findings in there. trufflehog git --since-commit scans diffs – a credential committed three years ago and never touched since is invisible to it, even though it’s sitting in the current checkout, which is the ordinary shape of a real leak. Every one of those fourteen came from the working-tree scan I added alongside it. Run both modes; if you only run one, run the filesystem scan.

The scheduled sweep had never scanned anything. For weeks the cron fired and died in preflight, because its default target pointed at a repository the App installation couldn’t read. Same lesson as above: a scanner that runs and finds nothing is indistinguishable from one that runs and does nothing.

Bigger runners buy nothing. Skipping archives cut scanned bytes by 79% but wall-clock time by only 12%, which localised the bottleneck precisely: walking git history is roughly 14× slower per byte than reading files from disk, and it does not parallelise. One core and twenty-four cores scan the same repository in the same time. Sharding across repositories is the only lever that works — worth knowing before anyone spends money on compute.

Never allowlist a directory, only a file. One entry I’d written as a directory path would have silently suppressed a cloud key that happened to share it. That key was verified live.

A verifier’s word is evidence, not proof. One detector in the suite returns verified for keys I fabricated by hand – I proved it by mutating characters and watching them pass. It’s excluded now. Given that verification is the main reason I chose the tool, that’s worth holding onto rather than filing away.

The biggest finding wasn’t a credential. Looking for who the scanner should notify, I found there was no answer: every organization member held admin on every repository I measured, across four samples. Any member could delete a repository, force-push over its history, or make a private one public, and the teams that existed were decorative – a grant of read can’t subtract an admin already held. That reframed my own advice. Moving credentials into Actions secrets is still worth doing, but a repository admin can add a workflow that prints any secret, so where base permission is admin an Actions secret is not the boundary it appears to be. I wrote that into the remediation guidance rather than letting the improvement read as a fix.

Where it landed

“Before” figures are raw scanner output — one finding per commit occurrence.

The 295 needs a caveat, because it was never 295 distinct problems. TruffleHog reports a finding for every commit that contains a credential, so one key committed forty times arrives as forty findings, and the same key living on two branches multiplies again. Grouping findings by credential instead of by commit collapsed 309 to 156 on its own, and criticals from 130 to 54. The rest of the drop came from reclassifying credentials no secret store could ever hide, like a maps key that ships to every browser.

So the honest reading of that table isn’t “we fixed 286 problems”. It’s that the estate had a handful of real leaks all along, and raw scanner output made them impossible to see. 295 findings is a number a team ignores; 9 is a number a team closes. These numbers cover the repositories the App reaches today – widening it across the full estate is an organization-owner action.

If there’s one thing to take from this: a scanner’s worst failure mode is not a false positive. It’s silence that looks like health.