Update Secrets (Secret Manager) #

Admin only. Secret values sit with GCP Admins.

Developers can list secret names but not read or write their values. roles/viewer grants secretmanager.versions.list and .get (metadata) but not secretmanager.versions.access (the payload), and not container.secrets.*, so the Kubernetes Secret is out of reach too.

Need a secret changed? Open a PR for the env-var mapping if one is needed, then ask in #back-end with the secret name. An admin sets the value.

Secret values change through the GCP CLI or the Cloud Console. They live only in Secret Manager — not in Git, a Helm values file, or a Terraform variable.

Publish, wait 2 minutes, restart. Takes about 5 minutes.

Publish the new value. Either path is fine:
```
gcloud secrets versions add <SECRET_NAME> --project=<project> --data-file=-
```
Paste the value, then Ctrl+D.

Or in the Console: Security → Secret Manager → pick the secret → New version. Same result; use whichever you trust more for a long or awkward-to-paste value.
Wait up to 2 minutes, then confirm the Kubernetes Secret picked it up:
```
kubectl -n fs-apps get secretsync backend-secrets \
  -o jsonpath='{.status.lastSuccessfulSyncTime}{"\n"}'
```
Wait for a timestamp after you published, then continue to step 3.
Restart so the pod re-reads its env. Manual today — see the TODO below:
```
kubectl rollout restart deployment/backend -n fs-apps
```

Verify without printing the secret. The two hashes must match:

kubectl exec -n fs-apps deploy/backend -- sh -c 'echo -n "$<ENV_VAR_NAME>" | sha256sum'
gcloud secrets versions access latest --secret=<SECRET_NAME> --project=<project> | shasum -a 256

Piping a value in #

Strip trailing newlines. A stored newline has broken hostnames before — 172.20.32.5\n:5432 produced a no such host error on the Grafana datasource.

gcloud sql instances describe ... --format='value(...)' | tr -d '\n' | \
  gcloud secrets versions add <SECRET_NAME> --project=<project> --data-file=-

Rotation updates the Secret, the restart loads it #

Secret Manager version N
  → CSI driver fetch                        ← automatic, ~2 min
  → SecretSync repacks `backend-secrets`    ← automatic, ~2 min
  → container env                           ← at container start only

The pod reads its config with envFrom: secretRef, which copies the Secret into the process environment when the container starts. A running process’s environment is fixed, so the new value arrives on the next start.

Wait for the sync, then restart. A restart before lastSuccessfulSyncTime advances loads the old value.

TODO: make the restart automatic. Step 3 is manual today. Two known options, neither evaluated yet:

Reloader (or a similar controller) watches backend-secrets and rolls the deployment when its content hash changes. No application change, one add-on to run.

Read from the mount. The backend reads /mnt/secrets-store/<NAME> instead of env and re-reads on change. Kubernetes updates volume-mounted secrets in place, so this removes the restart entirely — but it is a back-end change and touches every secret the app consumes.

Owner and target date unassigned. Raise in #back-end if secret rotation is costing you time.

Fix a sync that will not advance #

Confirm rotation is on:

kubectl get daemonset csi-secrets-store-gke -n kube-system \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.args}{"\n"}{end}'

Expect --enable-secret-rotation=true and --rotation-poll-interval=2m0s.

Read the object’s own status:

kubectl -n fs-apps describe secretsync backend-secrets

Rotation is set in Terraform at gcp/modules/gke, secret_manager_config.rotation_config.enabled.

Adding a brand-new secret #

One ops PR, then apply and add the value before you merge. A backend pod mounts every secret in backend.secrets before the app starts, so each one needs a value first.

Open the PR. Add the name in three places:
- backend_secret_ids in gcp/envs/staging/secrets.tf
- backend_secret_ids in gcp/envs/production/secrets.tf
- backend.secrets in k8s/charts/funnelstory/values.yaml
The chart list is shared by every env. The Terraform Plan parity check fails if the production list and the chart list differ.
Expect Helm CI / secrets to fail. It lists each secret with no enabled version in staging or production. Do not merge yet.
Apply Terraform from the PR branch. Staging first, then production. See Terraform. This creates an empty secret and gives the backend read access.
Add the value in both projects. Step 1 at the top of this page.
Re-run the failed secrets jobs. Merge when they pass. The merge deploys staging and production.

The deploy runs the same check before helm upgrade, and stops in seconds if a secret has no value. To force a deploy, start Helm Deploy (staging) or Helm Deploy (production) by hand and tick skip_secret_check.

Removing a secret #

Remove the name from all three places in one PR.
Merge, wait for the deploy, then apply Terraform.
Never delete a secret by hand while it is in backend.secrets. New pods stop starting at once.