Deploying a New App

1. Containerize the app

Start on the app side: make it a container the platform can run on GKE Autopilot. The image is built by the platform's Cloud Build (so it is automatically Binary Authorization-attested) and pushed to {app}-images. Autopilot enforces a hardened runtime contract, and the chart's Deployment pins a non-root securityContext — so the image has to actually run as UID 1001.

• Run as non-root UID/GID 1001 — the chart sets runAsNonRoot: true, runAsUser/runAsGroup/fsGroup: 1001. Create that user in the Dockerfile (e.g. adduser -D -u 1001 app) and USER it.
• No elevated privileges: the container runs with allowPrivilegeEscalation: false, all Linux capabilities dropped, and seccompProfile RuntimeDefault. Don't rely on root-only paths or extra caps.
• Listen on the port the chart targets — 3000 for web apps (containerPort name: http) — and bind to all interfaces (0.0.0.0, e.g. via the HOSTNAME env var), not just localhost, or the cluster can't reach the pod. Honor the PORT env var.
• Use tini (or another real init) as PID 1 for correct signal handling and zombie reaping.
• Set resource requests/limits — Autopilot bills and schedules on requests; they are required, not optional.
• Keep it small and production-only (multi-stage build, slim base such as node:22-alpine, NODE_ENV=production).

FROM node:22-alpine
WORKDIR /usr/server/app
RUN apk add --no-cache tini
RUN adduser -D -u 1001 app

COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
ENV NODE_ENV=production

USER app
EXPOSE 3000
ENTRYPOINT ["/sbin/tini"]
CMD ["npm", "run", "start"]

docker build -t {app}:local .
docker run -d --name {app}-test -p {host-port}:3000 {app}:local
curl -s -o /dev/null -w "%{http_code}\n" localhost:{host-port}/
docker exec {app}-test id -u          # expect 1001, not 0
docker rm -f {app}-test

2. Structure the app repository

Alongside the Dockerfile from step 1, add a workload Helm chart (depending on the shared gke-tenant-workload chart) with per-environment value overrides, plus the Cloud Build definitions for the dev, qa, and preview flows.

{app-repo}/
├── Dockerfile
├── helm/{app-name}/
│   ├── Chart.yaml               # depends on gke-tenant-workload
│   ├── values.yaml
│   ├── values/{dev,qa,preview,prod}.yaml
│   └── templates/
└── deploy/
    ├── cloudbuild/{dev,qa,preview}.yaml
    ├── skaffold.yaml            # used by Cloud Deploy's render phase
    └── k8s/reposync-{env}.yaml

3. Write the Helm chart

A Helm chart is a templated bundle of Kubernetes manifests — it declares how your app runs on the cluster. You don't write it all from scratch: your chart declares a dependency on the shared gke-tenant-workload chart, which supplies the platform-standard pieces (ServiceAccount + Workload Identity, the gateway ListenerSet/Certificate, ExternalSecrets, KEDA autoscaling). You add a thin chart — a Deployment, Service and HTTPRoute plus one values file per environment — and inherit the conventions. Nobody runs `helm install` by hand: at build time Cloud Build packages the chart and pushes it as an OCI artifact to the config-sync registry (oci://us-central1-docker.pkg.dev/u2i-bootstrap/config-sync); the RepoSync manifest points Config Sync at that chart + version, and Config Sync pulls it and applies it into your namespace.

• Declare the gke-tenant-workload dependency (pulled from the u2i-bootstrap/helm-charts OCI registry via `helm dependency build`; CI does the same at build time).
• Set the identity values under global: — appName, namespace, environment, stage, projectId, clusterName, clusterLocation, gatewayMode — in every environment's values file.
• Provide hostnames (the public URL) and serviceAccount.enabled: true, and enable a certificate (clusterIssuer for eg, certMap for managed-lb).
• Choose gatewayMode per environment: eg (in-cluster Envoy, nonprod dev/qa) or managed-lb (Cloud LB, prod).
• Add the templates the subchart doesn't render: a Deployment (with the non-root securityContext matching your image), a ClusterIP Service, and an HTTPRoute using the gke-tenant-workload.parentRef helper.
• Enable KEDA scale-to-zero in nonprod (a workloads entry with keda.maxReplicas + scaledownPeriod) so idle envs cost nothing; leave it off in prod (keda.enabled: false) so it stays always-on.
• When KEDA manages a workload, do NOT also set a static replicas on the Deployment — gate it out, otherwise Config Sync keeps resetting the count and fights the autoscaler.
• Keep one values file per environment under values/ (dev, qa, preview, prod).

# Chart.yaml — depend on the shared platform chart
dependencies:
  - name: gke-tenant-workload
    version: 0.11.2
    repository: oci://europe-west1-docker.pkg.dev/u2i-bootstrap/helm-charts

# values/dev.yaml — the platform contract lives under the subchart key
gke-tenant-workload:
  global:
    appName: {app}
    namespace: {app}-dev
    environment: dev
    stage: dev
    gatewayMode: eg            # managed-lb in prod
    projectId: c-u2i-nonprod
    clusterName: u2i-nonprod
    clusterLocation: europe-west1
  hostnames: [dev.{app}.u2i.dev]
  serviceAccount: { enabled: true }
  certificate: { enabled: true, clusterIssuer: letsencrypt-prod }
  workloads:                   # KEDA scale-to-zero (nonprod only)
    - name: {app}
      serviceName: {app}
      port: 80
      keda: { maxReplicas: 1, scaledownPeriod: 600 }

gcloud auth print-access-token | \
  helm registry login europe-west1-docker.pkg.dev -u oauth2accesstoken --password-stdin
helm dependency build helm/{app}
helm template {app} helm/{app} -f helm/{app}/values/{env}.yaml

4. Register the app in the foundation config

Now switch to the infrastructure side. Add an entry for the app under the tenant's apps map in u2i-gcp-infrastructure/foundation/4-tenants/terramate.tm.hcl, then run `terramate generate` to regenerate the derived terraform.tfvars — commit both files. Only set approver_group if that Google group already exists; otherwise use null and wire the prod gate later (binding a non-existent group breaks the IAM apply) — the same pattern compensation and u2i-comp-portal use.

• App key: max 20 characters. Foundation generates a {key}-deploy-sa service account, and GCP caps service-account IDs at 30 chars, so 30 − len("-deploy-sa") = 20. (Example: u2i-gcp-platform-docs is 21 → one over; use gcp-platform-docs instead.)
• The app key need NOT equal the GitHub repo name — github_repo is a separate field (e.g. key retrotool-app, repo retrotool). Keep the key short and DNS-safe; it also becomes namespaces, registry names, and the hostname base.
• While you're in this repo, also add a google_artifact_registry_repository_iam_member granting {app}-ci reader on shared-images-nonprod's docker-images repo, in foundation/4-tenants/artifact-registry-access.tf. This isn't created automatically by the app entry above — every app that actually builds has its own hand-added copy of this grant. Skip it and the first build fails at step 0 pulling the shared build-lib image with an 'artifactregistry.repositories.downloadArtifacts denied' error — easy to add now, annoying to debug later.

{app-key} = {
  display_name    = "..."
  github_repo     = "{repo-name}"
  branch          = "main"
  create_ar_repos = true
  # gcp-{app}-prodsupport@u2i.com once that group exists; else null
  approver_group  = null
}

# foundation/4-tenants/artifact-registry-access.tf — not created by the
# entry above; add it in the same PR or the first build fails at step 0.
resource "google_artifact_registry_repository_iam_member" "shared_images_{app}_ci_reader" {
  project    = local.shared_images_project_id
  location   = local.shared_images_region
  repository = local.shared_images_repo
  role       = "roles/artifactregistry.reader"
  member     = "serviceAccount:${google_service_account.app_ci["{app-key}"].email}"
}

terramate generate                                  # regenerate terraform.tfvars
cd foundation/4-tenants
terraform init -backend=false -input=false && terraform validate

5. Apply the foundation change

Applying runs through two separate Cloud Build runs — never locally. It is worth knowing which build does what:

• PR opened → pr-terraform-validation fires and runs terraform plan, posting a terraform-plan/foundation check. It shows the diff and catches errors (this is what flags a too-long service-account name, an invalid value, etc.) — but it does NOT apply anything.
• Merge to main → a second build (terramate-docker-images) fires and creates a release on the terramate-infrastructure Cloud Deploy delivery pipeline.
• That release waits at a manual approval gate; approving the rollout is what actually runs terraform apply. So a green PR check means "the plan is valid", not "it's live".
• The apply then provisions, via foundation/4-tenants/app-cicd.tf: three Cloud Build triggers ({app}-dev-trigger, {app}-qa-deployment, {app}-preview-deployment), the {app}-ci and {app}-deploy-sa service accounts, Artifact Registry repos ({app}-images, {app}-cache) in nonprod and prod, and two Cloud Deploy pipelines ({app}-dev-pipeline, {app}-qa-prod-pipeline).
• A first-time apply for a brand-new app can fail on the preview trigger with "user does not have impersonation permission on ... {app}-ci@..." — this is IAM propagation lag, not a real permissions gap (the {app}-ci service account was created earlier in the same apply and hasn't fully propagated yet). Everything else in the apply has already been created; just retry.
• Retrying the failed job (gcloud deploy rollouts retry-job) needs the clouddeploy.operator role — approver alone (what you use to approve the rollout) is not enough. If you don't have it via PAM, the simplest fix is to re-run the merge trigger instead (gcloud builds triggers run terramate-docker-images --branch=main) to create a fresh release and approve that one; Terraform is idempotent so it only creates what's still missing.

# PR status and the terraform-plan check
gh pr checks {pr-number} --repo {org}/{repo}
gh pr view {pr-number} --repo {org}/{repo} --json state,mergeable,mergeStateStatus

# after merge: find the release + rollout it created
gcloud deploy releases list --delivery-pipeline={pipeline} --region={region} --project={project} --limit=3
gcloud deploy rollouts list --release={release} --delivery-pipeline={pipeline} --region={region} --project={project}
gcloud deploy rollouts describe {rollout} --release={release} --delivery-pipeline={pipeline} --region={region} --project={project}

# inspect the underlying Cloud Build run
gcloud builds describe {build-id} --project={project} --region={region}

# retry a failed job (needs clouddeploy.operator, not just approver)
gcloud deploy rollouts retry-job {rollout} --release={release} --delivery-pipeline={pipeline} \
  --region={region} --project={project} --phase-id={phase} --job-id={job}

6. Declare namespaces, identity & buckets in u2i-tenant-infra

The namespaces the app runs in, its Kubernetes-to-GCP Workload Identity bindings, and any GCS buckets are declared in the separate u2i-tenant-infra repo (reconciled by Config Sync), not by the app's own pipeline. Add a subchart under chart/charts/apps/ depending on gke-tenant-foundation (+ gke-tenant-storage if it needs buckets), and enable it in k8s/{env}/rootsync.yaml. If the app needs its own public hostname, this is also where its DNS subzone gets created (gke-tenant-foundation.dns.publicZone) — see the next step.

• If the app uses gke-tenant-workload's KEDA HTTP scale-to-zero (workloads[].keda, the requestRate trigger — the default for non-prod), its dev/qa namespaces must ALSO be added to platform.keda.allowedNamespaces in k8s/nonprod/rootsync.yaml. This is a completely separate mechanism from the app's own reposyncRbac.bindings above: it's a platform-level chart (chart/charts/platform/templates/keda-http-rbac.yaml) that grants the Config Sync reconciler permission to manage HTTPScaledObject/ScaledObject, and it's driven by its own hand-maintained namespace list, not by the per-app subchart you just wrote. Skip it and Cloud Deploy still reports the rollout as SUCCEEDED — but the namespace ends up with zero resources (no pod, no HTTPRoute), because the reconciler's whole sync aborts on a KNV2009 'polling for status failed: ... httpscaledobjects.http.keda.sh is forbidden' error. Filed as a platform improvement to fold this into the per-app ClusterRole so it can't be missed again; until then, add the namespace to that list by hand.

# no kubectl access to a private cluster? verify via the real GCP resources instead
gcloud iam service-accounts describe {app}-k8s@{project}.iam.gserviceaccount.com --project={project}
gcloud iam service-accounts get-iam-policy {app}-k8s@{project}.iam.gserviceaccount.com --project={project}
gcloud dns managed-zones describe {app}-u2i-dev --project={project}

# if you do have cluster access
gcloud container clusters get-credentials {cluster} --region {region} --project {project}
kubectl get namespace {app} {app}-dev {app}-qa
kubectl get rootsync u2i-infra -n config-management-system -o jsonpath='{.status.sync.commit}'

7. Delegate the app's DNS subzone from the root zone

A hostname is not automatic — it has to be declared. u2i-gcp-infrastructure's foundation/6-dns/main.tf owns the root u2i.dev zone and delegates each app's subzone to it via an NS record. That delegation reads the nameserver set from the app's live subzone (data.google_dns_managed_zone, not a hardcoded value) — so it can only be added after step 6 has actually created the subzone; the data source would fail to resolve otherwise. Also add the app's hostname to domainFilters in u2i-tenant-infra's rootsync-external-dns.yaml, so per-environment A records (dev./qa.*) publish automatically from each HTTPRoute. This step only matters for public reachability — verifying the pod itself comes up (next step) doesn't need it; kubectl port-forward works without any DNS in place.

# confirm the delegation record in the root zone
gcloud dns record-sets list --zone={root-zone} --project={dns-project} --name="{app}.u2i.dev."

# confirm it resolves publicly (not just internally)
dig +short NS {app}.u2i.dev
dig +short dev.{app}.u2i.dev @8.8.8.8   # empty until a real HTTPRoute exists (step 8)

8. Build & release: branch, tag, or PR

A push to main fires {app}-dev-trigger, which builds the image, pushes it to {app}-images, and creates a release in {app}-dev-pipeline (deploying to the {app}-dev namespace). Cutting a version tag (v*) fires {app}-qa-deployment, which creates a release in {app}-qa-prod-pipeline.

# watch the build the trigger just fired
gcloud builds list --project={project} --filter="substitutions.TRIGGER_NAME={app}-dev-trigger" --limit=3
gcloud builds log {build-id} --project={project}

# verify the pod without needing DNS
gcloud container clusters get-credentials {cluster} --region {region} --project {project}
kubectl get pods -n {app}-dev
kubectl logs -n {app}-dev deploy/{app} -f
kubectl port-forward -n {app}-dev svc/{app} {local-port}:80

9. Automatic qa/nonprod rollout, manual prod approval

Cloud Deploy rolls the release out to the qa (nonprod) target automatically. Promotion to the prod target is gated behind a manual approval from the approver_group configured in step 4. The {app}-deploy-sa service account executes the rollout against the target cluster.

• A Cloud Deploy rollout reporting SUCCEEDED only means it applied the RootSync/RepoSync manifest — not that Config Sync finished reconciling it. Config Sync polls and applies asynchronously afterward, on its own schedule, and can still fail (wrong RBAC, invalid CRD, etc.) well after Cloud Deploy has already declared victory. Always confirm separately: kubectl get reposync {name} -n {app}-{env} -o yaml and check status.sync.errorSummary — an empty {} means it's actually clean.
• Zero pods in the namespace is not automatically a problem. If the workload uses KEDA scale-to-zero (the requestRate trigger, default in non-prod), minReplicas is 0 and pods only appear once traffic arrives — kubectl get pods legitimately shows nothing between deploys. The way to tell 'scaled to zero' apart from 'never got created' is the same RepoSync errorSummary check above, not the pod count.

gcloud deploy rollouts list --release={release} --delivery-pipeline={app}-qa-prod-pipeline \
  --region={region} --project={project}
gcloud deploy rollouts approve {rollout} --release={release} --delivery-pipeline={app}-qa-prod-pipeline \
  --region={region} --project={project}
kubectl get pods -n {app}-prod
kubectl get reposync {app}-{env} -n {app}-{env} -o jsonpath='{.status.sync.errorSummary}'

10. Wire up secrets and storage as needed

Secrets are pulled from GCP Secret Manager via the External Secrets Operator, using Workload Identity — no keys are checked in. If the app needs a GCS bucket (e.g. for user uploads), provision it via gke-tenant-storage (step 6); the pod mounts it directly through the gcsfuse.csi.storage.gke.io CSI driver.

11. (Optional) PR preview environments

Opening a PR fires {app}-preview-deployment (deploy/cloudbuild/preview.yaml), which builds a preview image, spins up a dynamic {app}-pr-{N} namespace, and applies the manifests directly for a per-PR review URL. The preview values enable KEDA with a shorter scaledown (previews are idle almost always) and render a cross-namespace ReferenceGrant so the dynamic namespace's route can reach the KEDA interceptor — static envs get that grant from u2i-tenant-infra, but a per-PR namespace must bring its own. Closing the PR triggers cleanup of the namespace.

gcloud builds list --project={project} --filter="substitutions.TRIGGER_NAME={app}-preview-deployment" --limit=3
kubectl get pods -n {app}-pr-{N}
dig +short pr-{N}.{app}.u2i.dev @8.8.8.8