Where a problem is documented follows who owns it:
| Area | Owner | Covered in |
|---|---|---|
| GitHub, Actions, workflows, the provisioning forms | this repository | below |
| Terraform remote state | this repository | below |
| Integration and login with a cloud provider | this repository | below |
| A cloud’s own services and how a template configures them | the infrastructure template | Azure · AWS |
The split is deliberate. The platform provisions any archetype, so anything tied to the resources a specific template creates — health checks, DNS, certificates, per-cloud Checkov baselines — belongs beside that Terraform, not here. Everything the platform itself owns lives on this page, once, covering every cloud rather than being restated in each cloud’s setup guide.
GitHub, Actions and the provisioning forms
HTTP 403: Resource not accessible by personal access token
Returned by a provisioning form, scripts/trigger-provision-<cloud>.sh, or a
raw POST /repos/<owner>/<repo>/dispatches.
The token needs Contents: read and write (fine-grained PAT or GitHub App
installation token) or the classic repo scope. Actions: write sounds like
the right permission for triggering a workflow but is not accepted for
repository_dispatch — this is the most common cause of the 403.
That is the only scope needed. The token does not require any cloud or
template-repo permissions; once dispatched, the workflow uses the platform’s
GH_PAT for cross-repo work and OIDC for the cloud. See
Token requirements.
CI in the new app repo fails with denied: permission_denied: write_package
The container push to GHCR (docker push ghcr.io/<owner>/<repo>:<tag>) is
rejected even though the platform workflow set the new repo’s default
workflow permissions to write. Common causes, in rough order of frequency:
-
The CI workflow declares its own
permissions:block that omitspackages: write. The block replaces the default — it doesn’t merge with it. The workflow must include all the scopes it needs, e.g.contents: read,packages: write,id-token: write. -
The login step uses the wrong token or username. For
docker login ghcr.io, expectusername: $andpassword: $— typos or a stale PAT will fail with the samedeniederror. -
Org-level setting overrides the repo setting. Org admins can lock workflow permissions at Settings → Actions → General with override disabled. The repo-level PUT is silently ignored. Ask the org admin to allow per-repo overrides or set the org default to
write. -
Image namespace mismatch. GHCR only accepts pushes to
ghcr.io/<owner>/<name>where<owner>matches the repo owner. A tag computed against a different org/user is rejected. -
A pre-existing GHCR package linked to a different repo (or unlinked). If a package with the same name already exists in the owner’s namespace from a deleted repo or earlier experiment, GHCR refuses pushes from this repo even with correct permissions. Visit
https://github.com/orgs/<owner>/packages(or/users/<owner>/packages), open the package’s settings and either delete it or use Manage Actions access to link it to the new repository.
Useful diagnostic command:
gh run view <run-id> -R <owner>/<app> --log-failed
Checkov reports new findings after a Terraform change
The checkov job gates every provisioning run and is scanned per environment:
prod against the template’s .checkov.yaml, dev/staging against its
.checkov.nonprod.yaml. A finding that appears only on prod is usually
intentional — the non-prod config skips a handful of prod-only baselines.
Either fix the finding or, if you’ve judged it a false positive or
not-applicable, add a justified entry to the template’s Checkov config
documenting why the check is skipped. See
CONTRIBUTING.md for the rules around skips, and
the template’s own troubleshooting for which checks it already skips and why.
Findings are uploaded to the repository’s Security tab under a per-cloud,
per-environment category (checkov-terraform-<env> for Azure,
checkov-terraform-aws-<env> for AWS) so results from different clouds never
overwrite each other.
A scheduled drift run does nothing, or reports less drift than expected
schedule events cannot carry inputs, so scheduled runs read their parameters
from repository variables (DRIFT_AZURE_*, DRIFT_AWS_*). A missing required
variable fails the run fast with an explicit error rather than silently doing
nothing.
Note that the cron expression itself cannot be driven by a variable:
GitHub does not evaluate $ anywhere in the on: block, so a templated
schedule is parsed as a literal, malformed cron and never fires. It has to be
a hard-coded string in the workflow file.
Under-reporting is a different failure. Where a template gates resources on an
input being non-empty — AWS’s main_domain, which guards the ACM certificate
and Route 53 record — passing an empty value makes the refresh-only plan skip
those resources entirely and report less drift than exists. Those inputs are
therefore mandatory even though the underlying Terraform variable has a
default. A wrong value fails loudly, which is the intended trade.
Environments exist but OIDC still fails
Check the environment name matches exactly — the OIDC subject is
repo:<org>/<repo>:environment:<env>, and Dev will not match dev.
See Cloud integration and login for the rest
of the trust configuration.
Terraform state
The platform owns the state backend for every cloud: one bucket or storage
account per application, bootstrapped by
scripts/bootstrap-tfstate-<cloud>.sh and safe to re-run. Symptoms below are
grouped by cloud only where the error text differs.
tofu init cannot find the backend
AWS — NoSuchBucket. The bucket is named
tf-state-<app20>-<account8>, where <app20> is the app name lowercased with
_ stripped and truncated to 20 characters, and <account8> is the first 8
characters of the AWS account ID. The formula is duplicated in three places
that must agree: scripts/bootstrap-tfstate-aws.sh, detect-drift-aws.yml
(which resolves the account via STS after login) and
provision-infrastructure-aws.yml (which derives it from field 5 of the role
ARN, with no STS call). If init can’t find the bucket, either the bootstrap
job didn’t run, or it ran against a different account than plan/apply did —
that is, the role ARN and the credentials in play disagree.
Azure — Error refreshing state. Most often a missing
Storage Blob Data Contributor assignment on the service principal; see the
next entry. If RBAC is correct, confirm the workflow passes use_oidc=true
and use_azuread_auth=true in its -backend-config flags (it does, by
default).
AuthorizationFailed during bootstrap-tfstate (Azure)
The service principal lacks one of the three required RBAC roles, or
propagation hasn’t finished. Re-run after a minute. If it persists, re-run the
az role assignment list command from step 3 of the
Azure setup guide and confirm all three roles are listed at
subscription scope.
Failed to query container 'tfstate' on '<account>' (Azure)
scripts/bootstrap-tfstate-azure.sh traps this on the
az storage container exists call. Two possible causes:
- RBAC: the SP has
Contributor(control plane) but notStorage Blob Data Contributor(data plane). Re-check step 3 of the Azure setup guide. -
Network rules: the storage account has
defaultAction = Deny(created by an earlier version of the script, or modified manually). The GitHub-hosted runner has no fixed egress IP and is blocked. Fix:az storage account update \ --name <account> --resource-group <rg> --default-action AllowThe current bootstrap script keeps
defaultAction = Allowby design — see the security-model note in step 3.
Cloud integration and login
Every cloud uses the same mechanism: GitHub issues a short-lived OIDC token, the cloud accepts it if the token’s subject matches something it trusts. Almost every failure here is a subject that doesn’t match, so start by comparing the exact subject string on both sides.
Three subject shapes are in play. Confusing them explains most “it works for some jobs but not others” reports:
| Who is authenticating | Subject |
|---|---|
Platform jobs with no GitHub Environment binding (resolve-inputs, bootstrap-tfstate, trust registration) |
repo:<owner>/<platform-repo>:ref:refs/heads/main |
Platform jobs bound to an environment (plan, apply, verify) |
repo:<owner>/<platform-repo>:environment:<env> |
| The application repo’s own deploy workflow | repo:<owner>/<app>:environment:<env> |
The first two are part of the one-time cloud setup. Only the third is registered automatically, by the provisioning workflow, and only when an application template repo was supplied — an infra-only run never registers app-repo subjects.
Both clouds — the subject has @<numbers> in it (repos created after July 15, 2026)
On top of the three shapes there are two subject formats. Repositories
created (or renamed/transferred) after July 15, 2026 present immutable-ID
subjects — repo:<owner>@<owner-id>/<repo>@<repo-id>:… instead of
repo:<owner>/<repo>:… — and both Entra and IAM match subjects verbatim, so
a credential registered in one format never matches a token minted in the
other. The tell is the assertion subject quoted in the error containing
@<numbers>, e.g.:
AADSTS700213: No matching federated identity record found for presented assertion subject ‘repo:deors@4376867/test-webapp@1349999312:environment:dev’.
The provisioning workflows handle this themselves: they ask GitHub which
prefix the new app repo actually presents
(gh api repos/<owner>/<repo>/actions/oidc/customization/sub) before
registering its subjects, and the decommission workflows remove either
format. Where it can still bite you is the platform repo’s own manually
registered subjects: if you stand up a fresh platform repo after the cutover,
query its prefix with the same API call and register the four setup-guide
subjects with that prefix.
Azure — AADSTS70021 / AADSTS700213: No matching federated identity record found
The token’s subject doesn’t match any federated credential on the App Registration. Re-check:
- The repo slug is exactly
<org>/<repo>— case-sensitive. - The subject format matches: if the quoted assertion subject contains
@<numbers>, see the previous section — the repo presents immutable-ID subjects and the credential was registered in the legacy format. - For environment subjects, the GitHub Environment exists with the exact
name (
dev,staging,prod) and the job runs against it. - If you triggered the workflow from a branch other than
main, the branch-scoped credential won’t match. Either trigger frommainor add another federated credential for that branch.
Azure — federated credential already exists with a stale subject
The Federated credential · <env> job fails reporting that the credential
named <owner>-<app>-<env> already exists but holds a different subject than
the app repo now presents. App names are not reused, so this almost always
means the app repo was deleted and recreated (e.g. while troubleshooting
an initial provisioning): GitHub embeds an immutable repo ID in OIDC
subjects, so the recreated repo presents a new subject and the stale
registration can never match a token again.
The workflow deliberately never repoints or deletes an existing identity credential — that is an operator’s decision. The job summary shows both subjects side by side. Typically the fix is to remove the stale registration and re-run:
az ad app federated-credential delete --id <azure_client_id> --federated-credential-id <owner>-<app>-<env>
Azure — Insufficient privileges to complete the operation registering a federated credential
The configure-federated-credentials job calls
az ad app federated-credential create, which hits Microsoft Graph
(POST /applications/{id}/federatedIdentityCredentials). Two things are
required and people commonly stop after the first:
- The SP is an owner of its own App Registration
(
az ad app owner add …). - The SP has the
Application.ReadWrite.OwnedByGraph application permission with admin consent.
Without (2), even a fully-owning SP gets Insufficient privileges. Run the
two-step procedure in step 3 — Allow the SP to manage its own federated
credentials of the Azure setup guide. Step (2) requires a
directory-role admin (Global, Privileged Role, Cloud Application, or
Application Administrator) — in a corporate tenant this is usually an internal
request.
Azure — Insufficient privileges to complete the operation on azuread_app_role_assignment during apply
The template assigns the environment’s end-to-end test client to the
E2E.Access role of the sign-in app. That call (POST
/servicePrincipals/{id}/appRoleAssignedTo on Microsoft Graph) is not covered
by Application.ReadWrite.OwnedBy: the SP also needs
AppRoleAssignment.ReadWrite.All and Application.Read.All as Graph
application permissions with admin consent. Both are granted in step 3 —
Allow the SP to manage its own federated credentials of the
Azure setup guide, with the same az rest loop.
Azure — AuthorizationFailed on Microsoft.Authorization/roleAssignments/write during apply
Terraform tried to create a role assignment — on the standard setup that is
exactly one scenario: the app pulls its image from Azure Container
Registry with managed identity, and the webapp template grants the app’s
identity AcrPull on the registry. The baseline roles (Contributor +
Storage Blob Data Contributor) deliberately cannot create role
assignments.
Fix: add the constrained delegation grant from the setup guide’s Only if
your apps pull from Azure Container Registry with managed identity section
— Role Based Access Control Administrator with the AcrPull-only
condition from setup/azure-policies/. Do not assign
User Access Administrator to get past it: that reopens the
grant-anyone-anything power the baseline removed.
Verify plan attestation fails: tfplan has no plan attestation from this run
The apply job only consumes a plan whose provenance attestation was signed
by the provision workflow in the same run. Two causes:
- The
Attest planstep did not run or failed in theplanjob — check that job first. The usual reason is a permissions change: the workflow needsid-token: write,attestations: writeandartifact-metadata: write, and an organization policy can cap theGITHUB_TOKENpermissions below that. The fix is on the organization’s Actions settings, not in the workflow. found: …lists a different run — the artifact does not come from this run. That is the situation the gate exists for; re-run the whole workflow rather than re-running theapplyjob alone.
Private repositories need a plan that includes artifact attestations; the public platform repository does not.
Azure — Delete resource group fails with deletion … failed after N min: …ResourceGroupDeletionBlocked…
The teardown’s wait watches the deletion’s outcome, not only whether the
group still exists: when Azure Resource Manager rolls the group back to
provisioningState=Succeeded, the job reads the failed
resourceGroups/delete event from the activity log and stops with its
message. The usual cause is an orphaned entry in the ARM index — a resource
the index still lists but whose provider answers NotFound — which blocks
the whole group deletion.
Fix: recreate a resource with the same name and type in that group (a minimal one is enough), then delete it explicitly; the index entry is refreshed and a re-run of the teardown completes. If the message names a resource lock instead, remove the lock and re-run.
Long deletions on their own are not a failure mode any more: the wait
re-authenticates to Azure every 30 minutes with a fresh OIDC token
(azure/login’s original assertion cannot be redeemed after the first access
token expires, which used to surface as AADSTS700024 on slow deletions) and
gives up only after three hours, naming the group’s last known state.
AWS — tasks stop with CannotPullContainerError (401 / 403 from the registry)
The service never reaches a steady state and every stopped task reports
CannotPullContainerError with an unauthorized or forbidden status from the
registry. The image is in a private registry (for example a GHCR package of a
private repository) and ECS could not authenticate the pull. Check, in order:
aws_registry_secret_arnreached the run — an input,client_payloadkey, or thePROVISION_AWS_REGISTRY_SECRET_ARNrepository variable. With none of them the task definition carries norepositoryCredentialsand ECS pulls anonymously, which private packages refuse.- The secret’s value is
{"username": ..., "password": ...}and the token hasread:packagesand can read that package (the token owner must have access to the application repository or its package). - The secret is readable by the task execution role — the template grants
secretsmanager:GetSecretValueon the ARN it is given; a customer-managed KMS key additionally needsregistry_credentials_kms_key_arn.
A 404 from the registry is a different problem: the image or tag does not exist — the preflight check catches that before anything is created.
AWS — Not authorized to perform sts:AssumeRoleWithWebIdentity
Raised in the application repo’s deploy workflow, on
aws-actions/configure-aws-credentials — not in the platform repo. The IAM
role’s trust policy has no subject matching
repo:<owner>/<app>:environment:<env>.
Check, in order:
-
Is the subject actually there?
aws iam get-role --role-name <role> --query 'Role.AssumeRolePolicyDocument' - Does the subject format match? A repo created after July 15, 2026
presents
repo:<owner>@<owner-id>/<app>@<repo-id>:environment:<env>— a legacy-format entry (or arepo:<owner>/*wildcard) never matches it. See the format section above. - Did the
configure-oidc-trustjob run, or was it skipped? Skipped means it was an infra-only run. - Did it fail with “has no statement federating token.actions.githubusercontent.com”? Then the one-time AWS setup was never completed: the GitHub OIDC identity provider and the role’s trust statement must exist before the workflow can add subjects to them. The workflow deliberately refuses to invent a trust statement — getting that wrong is a security problem, not a convenience.
The role also needs iam:GetRole and iam:UpdateAssumeRolePolicy on
itself. Easy to miss, because every other part of the pipeline works without
them.
AWS — a trust subject registered earlier has disappeared
The assume-role policy is a single document, frequently shared by several applications, and registering a subject is a read-modify-write. Concurrent writers lose updates: last write wins.
Handled within one run — configure-oidc-trust sets max-parallel: 1, so an
app’s own dev/staging/prod serialise, and the teardown workflows do the
same. Not handled: two different applications provisioning or being
retired at the same time against the same role. Provision and retire apps that
share a role one at a time.
AWS — AccessDenied / not authorized to perform <action> mid-run under the scoped policy baseline
With the scoped policies from the setup guide (instead of a broad managed
policy), an authorization error in a previously green step means the
infrastructure template has evolved past the policy set — for example a new
service, a new IAM role name prefix, or a managed-policy attachment the
PlatformEngScopedIAM conditions don’t list yet.
The error names the exact action and resource. Extend the matching policy
document under setup/aws-policies/, then roll it out as a new default
version — no detach/reattach needed:
aws iam create-policy-version \
--policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/<policy-name>" \
--policy-document file:///tmp/<updated-policy>.json \
--set-as-default
Do not fall back to AdministratorAccess to get past a single denied
action; IAM keeps only five policy versions, so prune old ones with
aws iam delete-policy-version if the create is refused.