Your allowlist is where the misses come from

Before publishing three project repositories I ran a credential sweep over them. The first pass came back nearly clean and I nearly believed it. The second pass, with the noise filter removed, found a live Gmail application password, a JWT signing key, an encryption key, a database administrator password reused across four services, a runtime credential-encryption key that no human had ever typed, and my own mobile phone number.

The pattern in the first pass was fine. What went wrong was the part I added afterwards to make the output readable.

The bug in my own filter

The first sweep was roughly this:

grep -rInE "(password|passwd|secret|token|api[_-]?key)\s*[:=]\s*['\"]?[A-Za-z0-9!@#$%^&*_+-]{6,}" . \
  | grep -viE "example|placeholder|changeme|\\\$\{|_PASSWORD\b|dummy|sample"

The second grep -v exists because the raw output is unusable — hundreds of lines of CancellationToken cancellationToken, var token = ..., Bootstrap’s DATA_API_KEY, and every ${VAR} reference in every template. Filtering is not optional if a human is going to read the result.

But look at _PASSWORD\b. I put it there to drop lines like password: ${DB_PASSWORD} — a reference, not a value. It also silently drops:

- GF_SECURITY_ADMIN_PASSWORD=<a real password>

An environment variable whose name ends in _PASSWORD, assigned a literal. Which is not an edge case — it is the single most common way a credential appears in a Compose file. My filter was so specifically wrong that it removed the highest-yield line shape in the entire corpus.

That is the lesson, and it generalises past this one mistake: a secret scanner’s recall is decided by its exclusions, not by its pattern. The pattern is easy — credentials look like assignments. The exclusions are where judgement lives, and every one you add to reduce noise is a shape a real finding can now hide in.

The fix is not a better exclusion list. It is to stop filtering the pattern and start filtering the review: take the full noisy output once, triage it by hand once, and then record the triage as explicit per-finding entries — a .gitleaksignore keyed on fingerprints — rather than as a broader regex. An allowlist of specific known-benign findings can only ever hide the things you looked at. A broadened pattern hides things you have never seen.

Search for values, not keys

The single highest-yield move in the whole audit had nothing to do with patterns. Once the second pass surfaced one password, I stopped looking for keys and started grepping for that literal string.

It appeared eight times, in three unrelated systems: as the Grafana admin password, as the MQTT broker password, as the SQL Server credential — and, helpfully, in the README, under a heading explaining how to log in.

Key-based search finds the first instance of a secret. Value-based search finds its blast radius. They answer different questions and you need both, in that order:

# 1. what is a credential?           (patterns, noisy, needs triage)
# 2. where else does *this* appear?  (literal, exact, exhaustive)
git grep -I -n -F -e '<the value>' $(git rev-list --all)

The second form over rev-list --all is also the only way to answer “is it gone”, because a working tree that is clean says nothing about history. A credential removed in a later commit is still served by every clone.

Password reuse is what turns one leak into a breach. One string, eight files, three systems: revoking it means touching all three, and finding all three means searching for the value.

Seven shapes, and what each one needs

Triaging the full output produced a taxonomy that I have found more useful than any pattern list. Roughly in order of how well tooling handles them:

1. Direct assignment. ADMIN_PASSWORD=hunter2, "Token": "abc123". Every scanner finds these. → Move to an environment variable, ship a .env.example, gitignore the real file.

2. Embedded in a compound string. This is where the tooling starts to slip:

Server=db,1433;Database=app;User Id=sa;Password=<value>;TrustServerCertificate=True;
postgres://user:<value>@host:5432/db?sslmode=disable

The key and the value are inside a longer literal, and the delimiter is ; or : or @ rather than a line-level =. A pattern anchored on “key, separator, value, end of token” misses it. → Either substitute the whole string from the environment, or assemble it at runtime from separate variables.

3. A default parameter in code.

ADMIN_TOKEN = os.environ.get("RBAC_ADMIN_TOKEN", "admin-demo-token")

Reads as good defensive practice and passes review for exactly that reason. A default credential that works is the reason nobody ever sets the real one — nothing ever tells the operator the system is running on a published string. → No default. Raise on startup and name the variable.

4. Generated runtime state, committed by accident. A Node-RED .config.runtime.json carrying _credentialSecret — the key that decrypts every stored credential in that instance. Also: an InfluxDB CLI config, a .terraform directory, an IDE workspace file. Nobody wrote these. A tool wrote them and git add -A swept them up. → These are the strongest argument for a .gitignore that is written while the tool is introduced, not after. And when auditing, git ls-files | grep -E '^\.|/\.' finds the dotfiles nobody remembers adding.

5. Development and scratch artefacts. An appsettings.Development.json with real values while the production appsettings.json was correctly templated. A .http request file containing a genuinely issued bearer token, pasted in to test an endpoint and committed with everything else. → The one that stung: whoever templated the production config did the right thing and then stopped, because dev config does not feel like production. The credential does not care how the file is named.

6. Personally identifying information that is not a credential. A personal mobile number as an SMS sandbox value, in three files. Personal email addresses as alert recipients. No secret scanner will ever flag these, and they are in some ways worse: you can rotate a password in a minute and you cannot rotate a phone number at all. → Worth an explicit pass, by hand, looking for your own details rather than for secrets.

7. Hashes rather than plaintext. A Traefik dashboard basicAuth entry — a bcrypt hash, so not directly usable. At cost factor 5 it is also cheap to attack offline, and the username is right there. → Low priority, non-zero, still belongs in an environment variable.

And the category that causes all the trouble:

8. Credential-shaped and harmless. changeme. rancher-bootstrap-change-me. An ANSIBLE_VAULT;1.1;AES256 ciphertext block, which looks exactly like high-entropy junk because it is. admin-demo-token. These are the false positives that make people write the exclusions that cause the misses in category 2 and 5. The loop closes on itself.

What changed, in practice

The fix pattern is boring and the same everywhere: real values into a gitignored .env, a committed .env.example documenting the shape, and the config files referencing variables. Compose reads .env automatically, so nothing about the local workflow changed — which matters, because a security fix that makes the day-to-day worse gets reverted.

Two details worth stealing:

• Match the vocabulary that already exists. One repository already had a well-organised environments/*.env.example set for its production path; only the development configs leaked. Reusing those exact variable names, instead of inventing new ones, meant the fix produced no new concepts to document.
• Files that are pure local state get gitignored, not templated. A generated CLI config or a Node-RED runtime file has no business in the repository at all. Ship a .example if the shape is useful; ignore the real one.

Then the honest test — not “did the pattern stop matching”, but:

git grep -I -l -E '<value1>|<value2>|…' $(git rev-list --all) | wc -l

across every commit in every repository. Zero, or you have not finished.

The part I keep

I have written and reviewed secret-scanning configuration before. I still shipped a filter whose most specific rule deleted the most common finding. The reason is not carelessness — it is that the filter was written while staring at hundreds of lines of CancellationToken, optimising for a readable screen, with no feedback at all about what the filter was removing.

So the practice I have taken from this: when you add an exclusion, print what it excluded. One extra pipeline stage, run once, that shows the diff between the raw and filtered output. Every line in there is something you have decided not to look at, and deciding that is exactly the part that should require looking.

The tooling in the scanning pipeline I built after this runs all of it on every push now. But no pipeline would have caught the mistake in this post, because the mistake was in the configuration, and a scanner cannot tell you about the shape of finding you told it to ignore.