Terraform #

Admin only. Applying needs project write access, which the developer role set excludes.

TODO: automate this through a PR. Plan and apply run from a laptop today. The target is a workflow that plans on the PR and applies on merge, so this page becomes review guidance rather than a procedure.

A staging change takes about 20 minutes end to end. Plan and apply:

terraform -chdir=gcp/envs/staging init
terraform -chdir=gcp/envs/staging plan -out=staging.tfplan
terraform -chdir=gcp/envs/staging apply staging.tfplan

All GCP infrastructure lives in the funnelstory/ops repo under gcp/.

Install #

brew install terraform
terraform version

Install a 1.9.x release. CI pins 1.9.* and every root declares required_version = ">= 1.9". The google provider is pinned to ~> 7.0 in each versions.tf. Bump it in its own PR, not alongside a feature.

Authenticate #

gcloud auth login
gcloud auth application-default login
gcloud config set project funnelstory-infra-staging

Terraform reads the application-default credential, not the gcloud auth login one. Both are needed. See gcloud setup.

Which directory hits which environment #

There are no Terraform workspaces. One directory per environment, one state per directory. The directory you run in decides the environment you change.

gcp/
├── bootstrap/          # one-time per project: state bucket, API enablement, CI identities
├── modules/            # shared: network, cloud-sql, gke, secret-manager, vpn, tunnel, kms
└── envs/
    ├── staging/
    └── production/

Two more roots live outside gcp/:

• cloudflare/envs/{staging,production} — tunnels, private-network routes, Zero Trust Gateway policies.
• grafana/envs/{staging,production,prod} — shared dashboards and alerting. prod is the legacy AWS root and stays until GCP production Grafana finishes its bake period.

Initialize #

terraform -chdir=gcp/envs/staging init

The backend is declared in versions.tf, so envs/* roots need no flags.

bootstrap/ is the exception. It is shared code with a per-environment backend and var file, so it needs both:

terraform -chdir=gcp/bootstrap init -backend-config=bucket=funnelstory-tf-state-staging
terraform -chdir=gcp/bootstrap apply -var-file=staging.tfvars

bootstrap/ state is remote, in the environment’s own state bucket.

Verify the target before you apply #

The plan output names the project on every resource. Read it — that is the check.

Use -chdir rather than cd, so the environment stays visible in the command you run and in your shell history.

Plan and apply #

Always save the plan to a file and apply that file. A bare terraform apply re-plans against state that may have moved since you read the output.

1. Plan to a file:

   terraform -chdir=gcp/envs/staging plan -out=staging.tfplan
   

2. Read the summary line. Expect N added, M changed, 0 destroyed.

3. Apply that exact file:

   terraform -chdir=gcp/envs/staging apply staging.tfplan
   

Any unexpected destroy is a stop signal. The GKE control-plane endpoint subnet and the Cloud SQL instances cannot be rebuilt without an outage.

For a narrow change, scope it:

terraform -chdir=gcp/envs/staging plan -target=module.database -out=db.tfplan

Production applies #

Production is manual and deliberate. CI runs fmt -check, validate, and plan only, with a read-only identity. There is no apply workflow.

Before you apply:

1. Confirm the active project is funnelstory-infra-production.
2. Confirm the plan has zero destroys and touches only what you intend.
3. Merge any other open branch that already applied to real state. Its resources read as “not in configuration” from your branch, and your plan will propose destroying them.
4. Post the plan summary in the PR or in #back-end.
5. Apply the saved plan file.

terraform -chdir=gcp/envs/production plan -out=production.tfplan
terraform -chdir=gcp/envs/production apply production.tfplan

What Terraform does not own #

Check this before you debug an apply that “did nothing.”

Terraform does own: projects and IAM, API enablement, VPC and NAT, GKE clusters, Workload Identity, Artifact Registry, Cloud SQL instances, GCS buckets, KMS keys, bastion hosts, monitoring and alerting.

Conventions #

1. Pin every provider in versions.tf.
2. Per-environment terraform.tfvars. No workspaces.
3. Keep credentials and secret values out of the repo.
4. Run terraform fmt before you push. CI fails on unformatted files.
5. Log every manual, non-Terraform action in docs/MANUAL_STEPS.md — date, environment, exact command, and whether production needs it too. That log is how a production cutover replays what staging did.

Gotchas #

Some resources have ignore_changes and Terraform will not correct them. The Cloud SQL module ignores backup_configuration, disk_size, and replica_configuration, because DMS and disk autogrow mutate them outside Terraform. Re-enabling production backups after promote is a gcloud sql instances patch, not an apply — see backup and restore.

Enabling an API and using it in the same apply can fail. GCP takes time to propagate API enablement. Apply bootstrap/ first, then the environment root.

CI plans production without the gitignored vpn.secret.auto.tfvars. The committed vpn.auto.tfvars topology plus try() keep that plan safe, so it does not propose destroying the live VPN tunnels. A local plan behaves the same way.

Next: run terraform -chdir=gcp/envs/staging plan -out=staging.tfplan and read the summary line.