Cloud Security Index

Preventing S3 Infinite Logging Loops

Alexander Hose on August 22, 20267 min. read

Abstract

Amazon Simple Storage Service (Amazon S3) provides two distinct observability mechanisms for auditing bucket operations: S3 Server Access Logging and AWS CloudTrail S3 Data Events. While CloudTrail records IAM identity attribution for AWS API requests, S3 Server Access Logging delivers low-latency, HTTP-level request diagnostics including detailed HTTP status codes, byte turnarounds, TLS cipher suites, and bucket-owner access records without incurring per-event CloudTrail Data Event fees.

However, enabling S3 Server Access Logging without strict architectural isolation introduces one of the most catastrophic misconfigurations in cloud security: the infinite recursive logging loop. If a bucket is configured to deliver its access logs to itself, or if server access logging is enabled on a bucket designated as the target for central CloudTrail trails, every single log delivery triggers a new access event. This spawns an exponential chain reaction resulting in millions of micro-objects, API throttling, and severe billing escalation within hours.

This article details the mechanics of S3 recursive logging loops, defines a tag-driven selective logging framework, and provides production-tested Service Control Policies (SCPs) and Terraform baselines as codified in AWS-DATA-004: Secure S3 Audit Trails and CloudTrail Loop Prevention.

bash
+---------------------------------------------------------------------------------------------------+
|                     S3 SERVER ACCESS LOGGING: RECURSIVE LOOP VS. SECURE CENTRAL PATTERN           |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
|  1. THE FATAL RECURSIVE LOGGING LOOP (HIGH-COST INCIDENT)                                         |
|                                                                                                   |
|     +-----------------------------------------------------------------------+                     |
|     |  Target Bucket: s3://customer-data-prod (Logging Target: ITSELF)      |                     |
|     |                                                                       |                     |
|     |  1. User GET Object --------> [ s3:GetObject ]                        |                     |
|     |                                      |                                |                     |
|     |  2. Logging Engine Generates Log ----v                                |                     |
|     |     [ s3:PutObject: /s3-access-logs/2026-08-22-01-00.log ]            |                     |
|     |                                      |                                |                     |
|     |  3. Log Delivery Is Itself an S3 Access Event! <----------------------+                     |
|     |     (Triggers NEXT s3:PutObject Log Delivery -> Exponential Explosion)                      |
|     +-----------------------------------------------------------------------+                     |
|                                                                                                   |
|  2. SECURE CENTRALIZED LOGGING ARCHITECTURE (ISOLATED & GUARDED)                                  |
|                                                                                                   |
|     [Workload Account: Member]                    [Log Archive Account: Central]                  |
|     +----------------------------+                +------------------------------------------+    |
|     | Tagged: SecuritySensitivity|                | Central Log Bucket:                      |    |
|     | s3://sensitive-pci-data    |                | s3://central-s3-access-logs-123456789    |    |
|     |                            |                |                                          |    |
|     | Server Access Logging:     |---(Deliver)--->| Strict Rules:                            |    |
|     | Target = Central Bucket    |                | - Access Logging on ITSELF: DISABLED     |    |
|     +----------------------------+                | - CloudTrail Destination: SEPARATE BUCKET|    |
|                                                   | - Guardrail: SCP Denies PutBucketLogging |    |
|                                                   +------------------------------------------+    |
+---------------------------------------------------------------------------------------------------+

1. Anatomy of an Infinite Recursive Logging Loop

An infinite logging loop occurs when the act of writing an audit log generates a new event that must itself be audited. In AWS S3, this takes three distinct forms:

1.1 The Self-Targeting Bucket Loop

An administrator enables server access logging on s3://app-data-prod and specifies TargetBucket: app-data-prod.

  1. An application reads an object: GET /data.csv.
  2. The S3 logging engine batches this event and delivers a log object: PUT /s3-access-logs/2026-08-22-12-00-00-UUID.
  3. The delivery of 2026-08-22-12-00-00-UUID is an S3 REST.PUT.OBJECT API call on app-data-prod.
  4. The logging engine records this REST.PUT.OBJECT call and schedules another log delivery: PUT /s3-access-logs/2026-08-22-12-05-00-UUID.
  5. Each subsequent log delivery generates more log lines, creating an exponential proliferation of micro-files that grows indefinitely until logging is disabled.
bash
Time 0:   1 Application GET  -------------------> 1 Log Record Queued
Time +1m: 1 Log Delivery PUT ------------------> 1 Log Record Queued
Time +2m: 1 Log Delivery PUT ------------------> 1 Log Record Queued
... If application receives continuous traffic:
Time +10m: (N App Requests + N Log Deliveries) -> Exponential Cascade of S3 Objects

1.2 The CloudTrail Log Bucket Collision Loop

A central bucket s3://org-cloudtrail-logs-central is configured to receive organizational CloudTrail audit trails from 50 AWS accounts. If an administrator subsequently enables S3 Server Access Logging on s3://org-cloudtrail-logs-central:

  • Every management and data event delivered by CloudTrail from any member account triggers an S3 REST.PUT.OBJECT.
  • The server access logging engine records the CloudTrail write and writes an access log.
  • The access log write is recorded as another event.
  • The bucket experiences continuous write amplification, overwhelming S3 partition prefix limits and inflating data storage charges.

1.3 Financial and Operational Consequences

  • S3 Request Cost Explosion: S3 charges $0.005 per 1,000 PUT requests. In an unchecked loop with multiple active prefixes, millions of micro-log files are written per hour, accumulating substantial API request charges.
  • S3 Partition Throttling: S3 supports 3,500 PUT/POST/DELETE requests per second per prefix. A runaway logging loop can exhaust prefix throughput, causing 503 Slow Down throttling errors that disrupt production applications sharing the bucket.
  • Management Overhead: Cleaning up hundreds of millions of micro-objects from an S3 bucket requires running S3 Batch Operations or aggressive S3 Lifecycle expiration rules, both of which incur additional processing costs.

2. Architecture of a Secure Log Landing Zone

To achieve comprehensive audit visibility without risking recursive loops, organizations must decouple workload storage from log aggregation.

bash
+---------------------------------------------------------------------------------------------------+
|                         MULTI-ACCOUNT SECURE S3 LOGGING ARCHITECTURE                              |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
|  [ Workload Account: Production (111122223333) ]                                                  |
|  +-------------------------------------------------------------------------+                      |
|  |  s3://pci-transactions-prod                                             |                      |
|  |  - Tag: SecuritySensitivity = High                                      |                      |
|  |  - Server Access Logging: Enabled                                       |                      |
|  |    Target: s3://org-s3-access-logs-999988887777                         |                      |
|  |    Prefix: s3-access-logs/111122223333/pci-transactions-prod/           |                      |
|  +-------------------------------------------------------------------------+                      |
|                                    | (s3:PutObject from logging.s3.amazonaws.com)                 |
|                                    v                                                              |
|  [ Dedicated Log Archive Account (999988887777) ]                                                 |
|  +-------------------------------------------------------------------------+                      |
|  |  s3://org-s3-access-logs-999988887777                                   |                      |
|  |  - Bucket Policy: Restrict to logging.s3.amazonaws.com + SourceOrgID    |                      |
|  |  - Access Logging on ITSELF: STRICTLY DISABLED                          |                      |
|  |  - S3 Lifecycle: Transition to Glacier Instant Retrieval after 30 days  |                      |
|  |  - Object Lock: Compliance Mode (WORM) Enabled                          |                      |
|  +-------------------------------------------------------------------------+                      |
|                                                                                                   |
|  +-------------------------------------------------------------------------+                      |
|  |  s3://org-cloudtrail-logs-999988887777 (Completely Separate Bucket)     |                      |
|  |  - Server Access Logging: STRICTLY DISABLED (Enforced by SCP)           |                      |
|  +-------------------------------------------------------------------------+                      |
|                                                                                                   |
+---------------------------------------------------------------------------------------------------+

2.1 Central Log Bucket Policy

The central bucket in the Log Archive account must permit log deliveries from the S3 logging service across the AWS Organization while locking down write permissions with condition keys:

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3ServerAccessLogDeliveryFromOrg",
      "Effect": "Allow",
      "Principal": {
        "Service": "logging.s3.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::org-s3-access-logs-999988887777/s3-access-logs/*",
      "Condition": {
        "StringEquals": {
          "aws:SourceOrgID": "o-xxxxxxxxxx"
        }
      }
    }
  ]
}

3. Tag-Driven Selective Logging Strategy

Enabling server access logging on every temporary bucket, scratch space, and development container registry generates unnecessary log volume. A tag-driven selective logging strategy focuses compliance and forensic resources on sensitive workloads.

3.1 Sensitivity Classification Standard

Organizations should mandate standardized bucket tags enforced during IaC provisioning:

  • SecuritySensitivity = High
  • DataClassification = Confidential | Restricted | Regulated

3.2 Automated Compliance Audit Script

The following Bash script evaluates all S3 buckets in an account, verifying that high-sensitivity buckets have active server access logging while ensuring that CloudTrail archive buckets remain loop-free:

bash
#!/usr/bin/env bash
set -euo pipefail

echo "================================================================="
echo " AUDITING S3 SERVER ACCESS LOGGING AND LOOP PREVENTION"
echo "================================================================="

# 1. Fetch all bucket names
BUCKETS=$(aws s3api list-buckets --query "Buckets[].Name" --output text)

for bucket in $BUCKETS; do
  # 2. Check if bucket is a designated CloudTrail repository
  if [[ "$bucket" =~ cloudtrail-logs-.* ]]; then
    LOG_STATUS=$(aws s3api get-bucket-logging --bucket "$bucket" --query "LoggingEnabled" --output text 2>/dev/null || echo "None")
    if [ "$LOG_STATUS" != "None" ] && [ -n "$LOG_STATUS" ]; then
      echo "CRITICAL RISK: CloudTrail bucket $bucket has Access Logging ENABLED! Potential recursive loop!"
    else
      echo "COMPLIANT: CloudTrail bucket $bucket has Access Logging disabled."
    fi
    continue
  fi

  # 3. Check for SecuritySensitivity Tag on workload buckets
  SENSITIVITY=$(aws s3api get-bucket-tagging --bucket "$bucket" \
    --query "TagSet[?Key=='SecuritySensitivity'].Value" --output text 2>/dev/null || echo "None")

  if [ "$SENSITIVITY" = "High" ]; then
    LOGGING_TARGET=$(aws s3api get-bucket-logging --bucket "$bucket" \
      --query "LoggingEnabled.TargetBucket" --output text 2>/dev/null || echo "None")

    if [ "$LOGGING_TARGET" = "None" ] || [ -z "$LOGGING_TARGET" ]; then
      echo "NON_COMPLIANT: Sensitive bucket $bucket is missing Server Access Logging!"
    elif [ "$LOGGING_TARGET" = "$bucket" ]; then
      echo "CRITICAL RISK: Sensitive bucket $bucket is logging to ITSELF! Recursive loop active!"
    else
      echo "COMPLIANT: Sensitive bucket $bucket logs to $LOGGING_TARGET."
    fi
  fi
done

4. Preventative Landing Zone Enforcement (SCPs & Terraform)

To prevent misconfigurations across decentralized engineering teams, security guardrails must be enforced using AWS Organizations Service Control Policies (SCPs) and immutable Terraform modules.

The following SCP blocks any IAM principal (including AdministratorAccess) from enabling server access logging on designated CloudTrail buckets across the organization:

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAccessLoggingOnCloudTrailBuckets",
      "Effect": "Deny",
      "Action": [
        "s3:PutBucketLogging"
      ],
      "Resource": [
        "arn:aws:s3:::org-cloudtrail-logs-*",
        "arn:aws:s3:::*-central-cloudtrail-*"
      ],
      "Condition": {
        "Bool": {
          "aws:PrincipalIsAWSService": "false"
        }
      }
    }
  ]
}

Summary Architecture Checklist

  1. Never target a bucket to itself: Verify that TargetBucket != SourceBucket on all S3 logging configurations.
  2. Isolate CloudTrail storage: Strictly prohibit s3:PutBucketLogging on all CloudTrail log archive buckets via SCP.
  3. Use prefix partitioning: Structure log targets using s3-access-logs/<account-id>/<bucket-name>/ to prevent partition hot-spotting and maintain clear access boundaries.
  4. Enforce tag-driven governance: Apply SecuritySensitivity: High tags to mission-critical buckets and use AWS Config rules to monitor compliance automatically.

For the full machine-readable specification, detection rules, and remediation scripts, refer to:

šŸ‘‰ AWS-DATA-004: Secure S3 Audit Trails and CloudTrail Loop Prevention.

We value your privacy

We use analytics cookies to understand how visitors interact with our site and to improve the user experience. You can choose to accept or decline these cookies.