Update Secrets (Secret Manager) #

Admin only. Secret values sit with GCP Admins.

Developers can list secret names but not read or write their values. roles/viewer grants secretmanager.versions.list and .get (metadata) but not secretmanager.versions.access (the payload), and not container.secrets.*, so the Kubernetes Secret is out of reach too.

Need a secret changed? Open a PR for the env-var mapping if one is needed, then ask in #back-end with the secret name. An admin sets the value.

Secret values change through the GCP CLI or the Cloud Console. They live only in Secret Manager — not in Git, a Helm values file, or a Terraform variable.

Publish, wait 2 minutes, restart. Takes about 5 minutes.

Publish the new value. Either path is fine:
```
gcloud secrets versions add <SECRET_NAME> --project=<project> --data-file=-
```
Paste the value, then Ctrl+D.

Or in the Console: Security → Secret Manager → pick the secret → New version. Same result; use whichever you trust more for a long or awkward-to-paste value.
Wait up to 2 minutes, then confirm the Kubernetes Secret picked it up:
```
kubectl -n fs-apps get secretsync backend-secrets \
  -o jsonpath='{.status.lastSuccessfulSyncTime}{"\n"}'
```
Wait for a timestamp after you published, then continue to step 3.
Restart so the pod re-reads its env. Manual today — see the TODO below:
```
kubectl rollout restart deployment/backend -n fs-apps
```

Verify without printing the secret. The two hashes must match:

kubectl exec -n fs-apps deploy/backend -- sh -c 'echo -n "$<ENV_VAR_NAME>" | sha256sum'
gcloud secrets versions access latest --secret=<SECRET_NAME> --project=<project> | shasum -a 256

Piping a value in #

Strip trailing newlines. A stored newline has broken hostnames before — 172.20.32.5\n:5432 produced a no such host error on the Grafana datasource.

gcloud sql instances describe ... --format='value(...)' | tr -d '\n' | \
  gcloud secrets versions add <SECRET_NAME> --project=<project> --data-file=-

Rotation updates the Secret, the restart loads it #

Secret Manager version N
  → CSI driver fetch                        ← automatic, ~2 min
  → SecretSync repacks `backend-secrets`    ← automatic, ~2 min
  → container env                           ← at container start only

The pod reads its config with envFrom: secretRef, which copies the Secret into the process environment when the container starts. A running process’s environment is fixed, so the new value arrives on the next start.

Wait for the sync, then restart. A restart before lastSuccessfulSyncTime advances loads the old value.

TODO: make the restart automatic. Step 3 is manual today. Two known options, neither evaluated yet:

Reloader (or a similar controller) watches backend-secrets and rolls the deployment when its content hash changes. No application change, one add-on to run.

Read from the mount. The backend reads /mnt/secrets-store/<NAME> instead of env and re-reads on change. Kubernetes updates volume-mounted secrets in place, so this removes the restart entirely — but it is a back-end change and touches every secret the app consumes.

Owner and target date unassigned. Raise in #back-end if secret rotation is costing you time.

Fix a sync that will not advance #

Confirm rotation is on:

kubectl get daemonset csi-secrets-store-gke -n kube-system \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.args}{"\n"}{end}'

Expect --enable-secret-rotation=true and --rotation-poll-interval=2m0s.

Read the object’s own status:

kubectl -n fs-apps describe secretsync backend-secrets

Rotation is set in Terraform at gcp/modules/gke, secret_manager_config.rotation_config.enabled.

Adding a brand-new secret #

Three parts, three different paths. All three are needed for the secret to reach the pod.

Container name → Terraform. Add the secret to gcp/envs/<env> and apply. See Terraform.
Value → GCP CLI or Console. Step 1 at the top of this page.
Env var mapping → ops repo PR. Add it under backend.secrets in k8s/deploy/<env>/values.yaml, merge, deploy. See update env vars.

Next: run step 1 for your secret, then watch lastSuccessfulSyncTime.