Rolling out an AWS data perimeter

This is the second part of Building an AWS data perimeter. The first part explains the controls: service control policies (SCPs), resource control policies (RCPs), VPC endpoint policies and KMS key policies. This part tells you how to attach them without an outage.

Why the order matters

Every control in part 1 is a deny. A deny in an SCP or RCP applies to every account under the target as soon as you attach it. The organization has no staging environment. A wrong condition breaks production org-wide as fast as the policy propagates. Only the person who holds the management account credentials at that time can fix it. That is why the order matters more than the content.

There is no order and no pipeline. AWS evaluates every policy that applies to the request together, and any one of them can veto it with an explicit deny. The network condition is not a stage of its own: it lives inside the SCP, the RCP, the bucket or key policy, or the endpoint policy.

Build an audit mode from CloudTrail

SCPs and RCPs have no native audit mode, so you build one. Write the deny. Before you attach it, query CloudTrail (Athena over the trail bucket, or CloudTrail Lake) for every request it would have matched:

  • S3 calls where the caller’s org and the bucket’s org differ
  • requests from workload roles with no vpcEndpointId
  • AssumeRole calls from accounts outside the org list

AWS publishes Athena queries for this in its data-perimeter-helper repository. The results are your exception list. The list is always longer than you expected.

The queries need history. The free CloudTrail event history holds 90 days of management events only. S3 object reads and writes are data events, which trails do not log by default, and network activity events are opt-in too. If they were never on, the S3 half of your exception list is empty, and that looks exactly like a clean result. Turn them on weeks before you attach anything. Scope them to the buckets and endpoints in the perimeter. The exception list covers only the period when these events were on. Both event types are billed per event, and Athena bills for the data each query scans.

Roll out one dimension at a time

Take each step through a sandbox OU, then non-production, then production, before starting the next:

  1. The resource perimeter SCP. Your workloads rarely write to foreign buckets, so a mistake has a small blast radius.
  2. The identity perimeter RCP.
  3. The network perimeter. It blocks the most legitimate traffic from places you forgot. Examples: developer laptops, CI runners in a vendor’s cloud, and the CSPM tool that assumes a role from its own account. Also the support engineer at 2 a.m. with a console session.
Attach lowest-blast-radius first: the resource SCP, then the identity RCP, then the network perimeter.

Check regions before you attach the network perimeter. The org and identity conditions are global, but VPC endpoints are regional. A workload in a region without endpoints has no private path, so the perimeter denies it.

  • List the regions your workloads run in and confirm the endpoints exist in each.
  • Deny the regions you do not use with an SCP on aws:RequestedRegion.
  • Give each workload that cannot take the private path its own named exception. Do not widen the policy for everyone. Expect a few: an old service hard-wired to a public endpoint, or a client that does not populate the condition keys the policy reads.

Break-glass access

Any of these denies can lock out the people who must fix it. Build and test the break-glass path before the first deny reaches production. AWS’s guidance on emergency access gives the design. A dedicated emergency access account, used for nothing else, holds a few IAM users (one per emergency administrator). Each workload account has a pre-created emergency role that trusts the roles in the emergency access account. Create all of it in advance. In an emergency, the APIs you would use to create access may be the ones that are broken.

  • Keep it off your normal login path. The emergency users sign in directly, not through your identity provider. An IdP outage or broken federation then cannot lock them out too.
  • Exempt it by exact ARN. Carve the emergency roles out of the network perimeter with aws:PrincipalArn, never a tag. Anyone who can write the tag can grant themselves the exemption (see the tag failure below).
  • Protect the path. An SCP denies changes to the emergency roles, their trust policies and the emergency users. Only the security team is exempt, so nobody else can widen, delete or quietly reuse them.
  • Test it against the perimeter. Run the whole path in the sandbox OU with the denies attached, before production. This proves the exemption works.

Rolling back

  • Keep every policy version in the repository. A rollback then means you re-apply the last good version from the pipeline. You do not rewrite JSON by hand during an outage.
  • Decide before launch who may detach a policy, from which account, and at what times. The pipeline normally runs in the delegated administrator account. The management account is the fallback when the pipeline itself is broken.

Failures that recur

AWS-owned buckets. The resource perimeter SCP denies buckets that AWS owns, and your workloads read several of them. The SSM agent and its patching read aws-ssm-<region> and similar buckets. Instances pull packages from the Amazon Linux repository buckets, and EKS nodes fetch from amazon-eks. None of these buckets has an aws:ResourceOrgID that matches yours. The problem is not limited to S3. EKS add-on images live in AWS-owned ECR repositories, and patch baselines are AWS-owned SSM resources. If your SCP covers those services, extend the exceptions to them. Copy the exception list from the AWS examples repository. The list changes, so check it again each time you onboard a new AWS service.

STS and CI federation. The identity perimeter RCP on STS blocks your CI federation. GitHub Actions or any OIDC provider assumes your roles with AssumeRoleWithWebIdentity, and that caller is an external identity with no aws:PrincipalOrgID at all. Scope the RCP’s org-ID statement to sts:AssumeRole and the other STS actions in AWS’s example. AWS leaves out sts:TagSession and sts:SetSourceIdentity on purpose, because SAML and web identity role assumptions call them too. Leave web identity and SAML to the trust policies, where the sub and aud conditions control access.

Then only one developer’s trust policy stands between the whole GitHub issuer and your role. A sub pattern can be wide enough to match another organization’s repositories. IAM rejects only a bare wildcard, so it does not catch this mistake. AWS’s identity perimeter sample gives the RCP a separate job here, EnforceTrustedOIDCTenants. No federated caller can pass the org check, so it asks whether the token came from your tenant of the provider:

{
  "Sid": "EnforceTrustedOIDCTenants",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Resource": "*",
  "Condition": {
    "StringNotLike": {
      "token.actions.githubusercontent.com:sub": "repo:my-github-org/*"
    },
    "Null": {
      "token.actions.githubusercontent.com:sub": "false"
    }
  }
}

The Null line limits the statement to tokens from that provider, leaving a Google or Cognito token to its own trust policy. The companion statement, EnforceTrustedOIDCProviders, denies web identity calls from any provider not on your list. With both statements, a trust policy written as repo:* admits nothing outside your GitHub organization. The check sits above the trust policy, where the developer cannot loosen it.

Third-party cross-account roles. Vendors that assume roles into your accounts are outside your org. An identity perimeter with no aws:PrincipalAccount allow-list cuts off the backup vendor, the security scanner and the cost tool in one attachment. List every cross-account trust before you write the RCP. That list is also your list of confused-deputy candidates. Many customers trust the same vendor role. Any other customer who learns your role ARN can point that vendor role at your account. The trust policy prevents this only if it requires an sts:ExternalId that the vendor holds for you alone. Check every one.

Service-linked roles. An AWS service creates these roles to act for itself. They are exempt from SCPs and RCPs, and so is their trust policy. That is usually what you want. Sometimes it is a surprise, when a service keeps doing something you thought you had denied. Service principals are not exempt from RCPs or endpoint policies, which is why the aws:PrincipalIsAWSService carve-out exists.

The unprotected exception tag. Several carve-outs in part 1 key on a principal tag. These are the workload roles the network perimeter applies to (dp:include:network), the resource perimeter SCP exception (dp:exclude:resource), and the endpoint policy’s escape hatch. When a tag decides an exception, the tag is part of the policy. By default, anyone with iam:TagRole in the account can write it. A developer can tag their own role out of the network perimeter without a change to any policy document. No deny statement detects this. Lock the tag down:

  • An SCP denies any request that sets a dp:* tag key unless the caller holds the admin tag.
  • A matching RCP denies sts:TagSession for the same keys, so the exception cannot arrive as a session tag (SAML sessions excepted).

The over-broad deny. Someone writes "Action": "*" in a network perimeter SCP or bucket policy to be safe. An RCP does not accept a bare * as the action, so this failure cannot start there. Within the hour, every sts:GetSessionToken from a laptop fails. The DescribeRegions calls behind the console fail, and so does every human console session. You find out from help desk tickets, because no monitor catches it first. Deny the data-bearing actions and nothing else.

Verifying the perimeter

Only the first check proves the perimeter holds, and it needs nothing but a second account and a laptop.

  1. The probe.
    • From an account outside the org, with a deliberately permissive test bucket policy, try to read the bucket.
    • From a laptop, with a real workload role’s credentials, try the aws s3 cp from the leaked-key example in part 1.
    • From inside the VPC, through the endpoint, do the same and confirm that it succeeds.
    • Run your break-glass path once, to show the perimeter did not lock it out.
  2. CloudTrail on the denies. The access-denied message names the policy type that denied the request. Filter for “explicit deny in a resource control policy” to see who is hitting the perimeter. Expect a spike on the day each policy attaches. Triage it through sandbox and non-production before you connect any alert to it.
  3. An external-access scanner. The perimeter covers only the services behind it. A separate tool must keep listing resources whose own policies admit principals outside the org. IAM Access Analyzer does this at the organization level, and its external access analysis is free. It does not fold RCPs into its evaluation, so findings stay open after the RCP lands, tagged resourceControlPolicyRestriction: APPLICABLE. The findings without that tag are your real gaps. A posture scanner such as Prowler or a commercial CSPM answers the same question with its cross-account and public-access checks for S3 and KMS.

Where to start

Starting a team from zero:

  1. Keep the SCPs and RCPs in a repository only the security team can write to, applied by a pipeline in a delegated administrator account. AWS Organizations lets you delegate policy administration to a member account (SCPs are the documented example). SCPs and RCPs cannot constrain the management account. Each credential and pipeline you keep out of it is blast radius removed from the account you can least afford to lose. Run nothing in the management account.
  2. Treat every change as a production deploy with a review.
  3. Turn on data events for the buckets in scope, network activity events for the endpoints, and your external-access scanner before writing a single deny.
  4. Test the break-glass path before the first deny attaches.
  5. Resource perimeter first, network perimeter last.

If the accounts sit in an Organization you do not control, write down the required posture and give it to the team that controls it. Then test it with the same probes, as if nobody had applied it. You cannot see their policies, so the probes are the only way to check them.

Sources and further reading