GKE Diagnostics #

Find the pod, then read why it is unhappy.

kubectl --context=connectgateway_funnelstory-infra-staging_global_funnelstory-staging \
  -n fs-apps get pods
kubectl --context=connectgateway_funnelstory-infra-staging_global_funnelstory-staging \
  -n fs-apps describe pod <pod>

Name the context on every command #

Both clusters use fs-apps for the application and observability for Grafana, and the workload names match too. A command therefore looks the same against either environment, and the context is the only thing that decides which one you hit.

Pass the context that gcloud setup generated:

connectgateway_funnelstory-infra-staging_global_funnelstory-staging
connectgateway_funnelstory-infra-production_global_funnelstory-production

Run kubectl config get-contexts to see what you have. A context missing from that list means you have not run get-credentials for it yet — see Get credentials below.

The examples further down omit --context to stay readable. Add it, or confirm the current context first:

kubectl config current-context

What you can run #

Command	Developer (`roles/viewer`)	Admin
`get`, `describe`, `logs`	yes	yes
`top pods`, `get events`	yes	yes
`exec`	no	yes
`rollout restart`, `delete pod`	no	yes

roles/viewer excludes container.pods.exec and container.pods.portForward on purpose — both would expose secret env vars. If a command below returns PERMISSION_DENIED, that is the boundary, not a bug. Ask in #back-end.

Two related tasks live elsewhere:

Reading logs — use Grafana, or kubectl logs for one pod’s output. See debug backend logs.
Deploying a change — open an ops repo Helm PR. See update env vars.

Get credentials #

gcloud container fleet memberships get-credentials funnelstory-staging \
  --project=funnelstory-infra-staging
# production: funnelstory-production / funnelstory-infra-production

Each command writes the context named above and makes it current. Use fleet memberships get-credentials, as here — see gcloud setup for why clusters get-credentials is the wrong one.

Why a pod is not ready #

Describe it and read the Events section at the bottom first. It names the real problem: image pull failure, failed probe, insufficient quota, missing secret.
```
kubectl -n fs-apps describe pod <pod>
```
For a crash loop, read the previous container’s logs. The current container may have only just started:
```
kubectl -n fs-apps logs <pod> --previous
```

Check recent cluster events:

kubectl -n fs-apps get events --sort-by=.lastTimestamp | tail -30

These are Autopilot clusters — Google manages the nodes, so describe pod is your window into scheduling. A pod stuck pending is almost always a resource request the scheduler cannot satisfy, and Events says so.

Resource pressure #

kubectl -n fs-apps top pods

Admin: rollouts #

kubectl -n fs-apps rollout status deployment/backend --timeout=300s
kubectl -n fs-apps rollout history deployment/backend
kubectl -n fs-apps rollout restart deployment/backend

A restart re-reads env vars from the ConfigMap and the Kubernetes Secret. It does not pull a new Secret Manager version on its own — see update secrets.

Delete the pod rather than waiting out a CrashLoopBackOff once the fix is in. The backoff grows to about 5 minutes and will otherwise sit there long after the problem is solved:

kubectl -n fs-apps delete pod <pod>

Helm owns these deployments, so check the release when a rollout looks wrong:

helm -n fs-apps list
helm -n fs-apps history funnelstory
helm -n fs-apps get values funnelstory

Admin: confirm configuration reached the pod #

# non-secret env vars
kubectl -n fs-apps exec deploy/backend -- printenv | sort

# a secret — compare hashes rather than printing the value
kubectl -n fs-apps exec deploy/backend -- sh -c 'echo -n "$FS_DB_DSN" | sha256sum'
gcloud secrets versions access latest --secret=FS_DB_DSN \
  --project=funnelstory-infra-staging | shasum -a 256

Equal hashes mean the value reached the pod. Unequal means SecretSync or the restart did not complete:

kubectl -n fs-apps get secretsync backend-secrets \
  -o jsonpath='{.status.lastSuccessfulSyncTime}{"\n"}'

A stale timestamp means the Kubernetes Secret has not picked up a new Secret Manager version yet.

Admin: reach an internal Service #

kubectl -n fs-apps get svc
kubectl -n fs-apps port-forward svc/backend 8000:8000
curl -sS http://localhost:8000/api/ping

Services resolve in-cluster as <svc>.fs-apps.svc.cluster.local — for example http://highcharts.fs-apps.svc.cluster.local:7801.

Next steps #

Debug backend logs — history beyond the current pod
Grafana — rates and trends, no IAM needed
Get pprof dumps — profile one pod over the VPN

Next: run the get pods command at the top and check the RESTARTS column.