A Secret in a GitOps repo is a Secret in production

The SOC namespace in my thesis cluster is reconciled by Argo CD out of two directories, manifests/ and data/. That is the point of GitOps: the repository is the only thing that decides what runs. It is also the trap, because it means anything a manifest contains, the repository contains — permanently, for everyone with read access, in a repo whose entire design goal is that more people can read it.

Four of those manifests looked like this:

apiVersion: v1
kind: Secret
metadata:
  name: postgres-creds
  namespace: soc
type: Opaque
stringData:
  POSTGRES_USER: soc
  POSTGRES_PASSWORD: soc-demo-password

It is tempting to wave that away — demo values, a lab cluster, nobody cares. I think that instinct is exactly the problem, for a reason that has nothing to do with this password being weak. The kind: Secret resource is doing no work here. Kubernetes Secrets are base64, not encryption; their only real property is that they are a separate object with separate RBAC. Write the value into the manifest and you have thrown away the one thing the resource was for, while keeping the name that makes it look handled.

So: take them out. What follows is what that actually involved, including the part where I did it and it accomplished nothing.

Getting the Secrets out of the sync path

The mechanical bit is easy. Split every kind: Secret document out of the directories Argo CD watches, into a secrets/ directory it does not:

soc/
├── manifests/          ← Argo CD app zt-soc-core
├── data/               ← Argo CD app zt-soc-data
└── secrets/
    ├── soc.env.example   committed — the shape
    ├── soc.env           gitignored — the values
    └── bootstrap.sh      applies them, out of band

The script is kubectl create ... --dry-run=client -o yaml | kubectl apply -f - for each one, which is the idiom that makes secret creation idempotent — kubectl create secret alone fails on the second run, and re-running a bootstrap script has to be free or nobody will run it:

apply_secret() {
  local name="$1"; shift
  kubectl create secret generic "$name" -n "$ns" "$@" \
    --dry-run=client -o yaml | kubectl apply -f -
}

Five Secrets, one command, safe to repeat. Then I grepped the tree again to confirm the credentials were gone.

They were not.

The line that made all of it pointless

env:
  - name: DATABASE_URL
    value: "postgres://soc:soc-demo-password@postgres.soc.svc.cluster.local:5432/soc?sslmode=disable"

The postgres-creds Secret existed. The Deployment did not use it. The password was written into a plain value: in the pod spec, and the Secret sat alongside as decoration. Extracting the Secret changed nothing at all, because the Secret was never where the credential lived.

This is worth dwelling on, because the failure is not “someone forgot”. The Secret and the Deployment were both correct-looking in isolation. postgres-creds is a real Secret with real RBAC. The Deployment is a normal Deployment with a normal env var. Nothing about either file is obviously wrong, and no linter I had would flag the pair — you only see it by asking a question that spans two documents: does the credential appear anywhere other than the Secret?

That question, and not “do we have a Secret”, is the one worth automating.

An inline value: is not a slightly worse Secret. It is a value that appears in kubectl get deploy -o yaml, in kubectl describe pod, in the Argo CD UI’s live manifest view, in the diff Argo shows on every sync, and in the audit log of every update on that object. A secretKeyRef appears in all the same places as the name of a Secret.

Building the DSN at pod start

Postgres wants one connection string; the Secret has three fields. The kubelet will do the assembly:

env:
  # Assembled from postgres-creds rather than written out inline, so
  # the password never appears in a manifest, a `kubectl get deploy`
  # or a pod description.
  - name: POSTGRES_USER
    valueFrom:
      secretKeyRef: { name: postgres-creds, key: POSTGRES_USER }
  - name: POSTGRES_PASSWORD
    valueFrom:
      secretKeyRef: { name: postgres-creds, key: POSTGRES_PASSWORD }
  - name: POSTGRES_DB
    valueFrom:
      secretKeyRef: { name: postgres-creds, key: POSTGRES_DB }
  - name: DATABASE_URL
    value: "postgres://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@postgres.soc.svc.cluster.local:5432/$(POSTGRES_DB)?sslmode=disable"

Three details about $(VAR) expansion that are easy to get wrong:

• It is $(VAR), not ${VAR} and not $VAR. The other two are passed through as literal text, so your application receives a DSN with ${POSTGRES_PASSWORD} in it and fails to connect with an error that says nothing about env expansion.
• Order matters. A variable can only reference variables defined earlier in the same container’s env list. Put DATABASE_URL first and it expands to the literal string, silently.
• A literal $( must be escaped as $$(. Rare, but if your password can contain $( you have a different and more exciting problem.

The result is that the password exists in exactly one place in git — as the string POSTGRES_PASSWORD inside a secretKeyRef.

What this does not do, and it is worth being straight about: the value is still in the container’s environment at runtime, so it is readable by anything that can kubectl exec into the pod, by anything with node access reading /proc/<pid>/environ, and by any crash handler that dumps the environment. Moving from value: to secretKeyRef: raises the bar from “in the repo forever” to “requires access to the running cluster”. It does not make the credential secret from someone already inside. A projected file mount that the process reads on startup is the next rung; that is a larger change to the application than it is to the manifest.

The default credential in the application

Same audit, different shape:

ADMIN_TOKEN = os.environ.get("RBAC_ADMIN_TOKEN", "admin-demo-token")
BASIC_USER  = os.environ.get("ADMIN_USERNAME", "admin")
BASIC_PASS  = os.environ.get("ADMIN_PASSWORD", "admin")

Three lines that pass every review because they look like good defensive coding. They are the worst finding of the lot, and the reason is in the mechanism, not the values.

A default credential that works is precisely why nobody ever sets the real one. The deployment comes up. The console loads. Log in as admin/admin — it works, so it must be configured. There is no moment at which the system tells anyone that the token protecting its admin API is a string published in a public repository. The failure is silent by construction, and it stays silent for as long as the fallback keeps working, which is forever.

def _required(name: str) -> str:
    """No default. A fallback credential is still a credential, and a working
    default is the reason nobody ever sets the real one."""
    value = os.environ.get(name)
    if not value:
        raise RuntimeError(
            f"{name} is not set — apply the soc Secrets first (secrets/bootstrap.sh)"
        )
    return value


ADMIN_TOKEN = _required("RBAC_ADMIN_TOKEN")

Now a misconfigured deployment cannot start. Which brings up the objection I actually care about.

The failure mode is the design

With the Secrets out of the sync path, Argo CD reconciles the Deployments happily and the pods sit in CreateContainerConfigError until someone runs the bootstrap script. That looks like a regression. Before, kubectl apply -f manifests/ produced a running system.

It is not a regression, and stating why is the whole argument:

	before	after
fresh cluster, no bootstrap	runs, with a credential from a public repo	fails to start, naming the missing variable
someone reads the repo	has the production credential	has the name of a Secret
operator rotates the password	edit a manifest, commit, push, sync	edit soc.env, re-run one script
“is this configured properly?”	unanswerable by looking	answerable: the pod is running

The second row is the one that matters, and the last row is the one that makes the whole thing self-enforcing. A system that cannot start without real credentials is a system where “it is running” is evidence that someone supplied them. That is a much stronger property than any amount of documentation, and it costs one startup check.

Loud, immediate, at startup, before serving a single request, with the missing variable named in the error — that is the best failure available. The alternative is a console that quietly accepts a well-known token from anyone on the cluster network.

Where this actually ends

A bootstrap script is not the destination. It is the step that removes the credential from git today, without a cluster round-trip and without adding a dependency. The real answers are the ones that let the encrypted form live in git so the GitOps property is preserved:

• Sealed Secrets — the controller already runs in this cluster’s platform tier, which makes it the obvious end state. A SealedSecret is safe to commit and only that cluster can decrypt it. The cost is that sealing needs the cluster’s public key, so it is not something you can do offline before the cluster exists, and re-sealing per cluster is manual.
• External Secrets Operator with Vault — what I work with at my day job, and the right answer once there is more than one cluster and more than one operator. Rotation happens in one place and every consumer follows. The cost is that Vault has to be up, and highly available, and someone has to run it.
• SOPS with age or KMS — encrypted values in the manifest itself, decrypted at apply time. Good middle ground, no operator, but Argo CD needs a plugin to render it.

All three are better than a script. All three are also strictly more machinery, and none of them would have found the DATABASE_URL line — because that line was not a secrets management problem. It was a credential written somewhere nobody thought to look, next to a Secret that made everyone assume the question was already answered.

Which is the part I would keep if I could only keep one sentence: having a Secret is not the same as using it, and only a grep for the value itself can tell the difference.