Grafana

Grafana #

Open the link and log in with Cloudflare Access. No Grafana password, no database credentials.

Access #

  1. Open the link.
  2. Log in through Cloudflare Access with your @funnelstory.io or @funnelstory.ai account.

Cloudflare Access is the only gate. Grafana proxies every query, so you need no database credentials of your own.

Start here when something is wrong #

Symptom Dashboard What to look at
Database looks slow cloud-sql read ops/sec, CPU, disk I/O vs baseline
Errors, cause unknown gcp-operations narrow the time range, filter severity>=ERROR
API slow or failing backend request and error rates
Work not draining queues queue depth and drain rate
LLM cost or latency spike llm-telemetry-gcp call volume, latency, cost

“The database looks slow” #

Open cloud-sql. Check read ops/sec, CPU, and disk I/O against baseline.

A repeating burst on a fixed interval is usually a scheduled backend job, not an infrastructure fault. The prediction-scores sweep runs every 6 hours and has tripped the disk-I/O alert before — baseline is ~560 read ops/sec, the burst hits 2100–2760 for about 10 minutes.

If the burst is read-heavy on staging, confirm FS_DB_REPLICA_DSN points at funnelstory-staging-replica (172.21.32.13) and not at the writer.

“Something is failing and I cannot tell what” #

  1. Open gcp-operations and narrow the time range to the incident.
  2. Filter to severity>=ERROR.
  3. Move to backend for rates over time.
  4. Use the BigQuery panels to group errors by message.

“Is it one pod or all of them?” #

Grafana shows aggregate behaviour only. Confirm per-pod with GKE diagnostics.

Dashboards #

Eight dashboards ship from one shared source and are identical in both environments.

Folder Dashboard Use it for
Operations gcp-operations first stop for a GCP-side problem
Operations cloud-sql CPU, memory, disk I/O, connections
Operations backend backend request and error behaviour
Operations queues queue depth and drain rate
AI llm-telemetry-gcp LLM call volume, latency, cost
AI agent-analytics agent runs and outcomes
Business subscriptions subscription state
Business license-subscriptions license and seat counts

Edit dashboards in the ops repo, not in the UI. The JSON at grafana/dashboards/<slug>/<slug>.json is the source of truth. Grafana’s own database is runtime state, so a UI edit is lost on the next apply.

Datasources #

Both environments provision the same four UIDs, so a dashboard is portable between them.

UID Type Reads
funnelstory-db PostgreSQL the application database, as read-only monitoring
funnelstory-gcp-logs Google Cloud Logging GCP logs for that project
funnelstory-gcp-monitoring Google Cloud Monitoring Cloud SQL and infra metrics
funnelstory-bq-logs BigQuery Log Analytics views over the app log bucket

Dashboards reference these through datasource-typed template variables (${Database}, ${GCPLogs}, ${BigQuery}) rather than a hardcoded UID.

Why both a logging and a BigQuery datasource: Cloud Logging cannot aggregate. It returns log rows only, with no stats equivalent. Any panel that counts, sums, or groups reads the same logs through BigQuery. See debug backend logs.

Staging lists a second Postgres datasource with the random UID cfscsv22r3kzka. Use funnelstory-db.

Environment mapping #

Staging Production
GCP project funnelstory-infra-staging funnelstory-infra-production
GKE namespace observability observability
Helm release grafana-staging grafana-production
Metadata Cloud SQL instance grafana-staging grafana-production

Grafana metadata lives in its own Cloud SQL instance, separate from the application database. There is no SQLite volume, so a pod restart loses nothing.

Query the app database ad hoc #

From grafana/ in the ops repo:

./query "SELECT count(*) FROM accounts"                 # production Postgres
./query -s Kh5fT8-Vz "SELECT count(*) FROM accounts"    # staging Postgres

Needs a Grafana service-account token in grafana/.env. Read-only, through the monitoring role.

If Grafana itself is down #

  1. Check the pods. Expect two containers: grafana and cloud-sql-proxy.

    kubectl -n observability get pods

  2. Read both containers’ logs:

    kubectl -n observability logs deploy/grafana-staging -c grafana --tail=200
kubectl -n observability logs deploy/grafana-staging -c cloud-sql-proxy --tail=200

  3. If Cloudflare returns 524, port-forward and test internally first:

    kubectl -n observability port-forward svc/grafana-staging 33000:3000
curl -sS http://127.0.0.1:33000/api/health

A proxy authentication failure points at IAM. A healthy /api/health behind a 524 points at the Cloudflare tunnel, not at Grafana.

Full runbooks live in the ops repo: docs/GRAFANA_STAGING_RUNBOOK.md and docs/GRAFANA_PRODUCTION_RUNBOOK.md.

Next: open the staging link and load the gcp-operations dashboard.