26 of 50 in All Controls
aws

Centralized VPC Flow Logging with Partitioning and Compression

IDAWS-LOG-004
Severity
medium
Automation
Auto
Category
logging
Enforcementmandatory
Complexity / Impact
medium(none impact)
Risk Profile
medium likelihood(low residual)
Cloud Scopeaws:account
Environments
all
Target Resources
AWS::EC2::VPCAWS::EC2::FlowLogAWS::S3::Bucket
Services
vpcs3kmsconfigiam
Tags
networkloggingvpcflow-logsmonitoringparquetlanding-zone
Frameworks
CIS AWSNIST 800-53ISO 27001SOC 2PCI DSS
Governance
Cloud Security (quarterly)
Created
2026-05-03
Last Updated
2026-08-21

Specification & Description

Enable VPC Flow Logs across all Virtual Private Clouds (VPCs), subnets, and transit gateways, aggregating traffic telemetry into the central Log Archive S3 bucket formatted with Parquet compression and 1-minute aggregation intervals.

Technical Specifications

VPC Flow Logs capture network telemetry regarding IP traffic flowing to and from elastic network interfaces (ENIs) across all VPCs in the Landing Zone.

1. Multi-Account Network Telemetry Aggregation

  • Flow logs from every workload, transit, and shared services account must be published directly to the dedicated Log Archive account S3 bucket using the delivery.logs.amazonaws.com service principal.
  • S3 bucket policies restrict delivery to authorized organization IDs via aws:SourceOrgID.

2. Performance & Cost Optimization Best Practices

  • Storage Format: Store flow logs in Apache Parquet format rather than plain text. Parquet provides columnar compression, reducing S3 storage costs by up to 80% and speeding up Amazon Athena / SIEM query execution.
  • Hive-Compatible S3 Partitions: Enable Hive-compatible partitions (/year=2026/month=08/day=21/) for optimized query pruning and data lifecycle management.
  • Aggregation Interval: Enforce a 1-minute (60-second) maximum aggregation interval on production VPCs to quickly capture ephemeral port scanning and fast lateral movement attempts.

3. Governance via AWS Config

  • Automated AWS Config managed rules (VPC_FLOW_LOGS_ENABLED) detect any newly provisioned VPC that lacks active flow logging and trigger automatic remediation.

Security Rationale

VPC Flow Logs provide the essential network forensic layer for detecting unauthorized lateral movement, port scans, command-and-control (C2) beaconing, and data exfiltration. Centralizing flow logs in an optimized Parquet format ensures consistent data retention, rapid threat investigation, and cost-effective SIEM ingestion across all multi-account workloads.

Business & Security Impact

Without centralized VPC Flow Logs, security teams are blind to network-level communication, preventing forensic reconstruction of network intrusions, DDoS attacks, or anomalous internal traffic.

Compliance Framework Mappings (5)

Regulatory baselines and industry standards mapped to this control specification.

Beta
Framework / StandardMapped Requirements & ControlsControls
CIS AWS Foundations Benchmark v3.0CIS AWS
4.3
1
NIST 800-53 Rev 5NIST 800-53
AU-12CA-7SC-7
3
ISO 27001:2022ISO 27001
A.8.15A.8.20
2
SOC 2SOC 2
CC7.2
1
PCI DSS v4.0PCI DSS
10.2.11.2.4
2

* Compliance framework mappings are provided for informational architectural reference. Always verify control alignment against your organization's formal audit requirements.

Automated Detection & Rule Logic

Detection Engine: aws-api • Automated: Yes

Audits all VPCs across active regions to ensure that at least one active VPC Flow Log is enabled, traffic-type is set to ALL, and logs are delivered to the centralized S3 destination.

AWS CLI: List VPC Flow Logs Configurationbash
aws ec2 describe-flow-logs \
  --query 'FlowLogs[*].[FlowLogId, ResourceId, TrafficType, LogDestinationType, LogDestination, FlowLogStatus]' \
  --output table
Python (Boto3): Audit VPC Flow Log Coveragepython
import json
import boto3

ec2 = boto3.client('ec2')

def audit_vpc_flow_logs():
    vpcs = [v['VpcId'] for v in ec2.describe_vpcs().get('Vpcs', [])]
    flow_logs = ec2.describe_flow_logs().get('FlowLogs', [])
    
    monitored_vpcs = {
        fl['ResourceId']: fl for fl in flow_logs 
        if fl.get('FlowLogStatus') == 'ACTIVE' and fl.get('TrafficType') == 'ALL'
    }

    non_compliant = []
    for vpc_id in vpcs:
        if vpc_id not in monitored_vpcs:
            non_compliant.append(vpc_id)

    if non_compliant:
        return {"status": "NON_COMPLIANT", "unmonitored_vpcs": non_compliant}
    return {"status": "COMPLIANT", "monitored_vpcs": list(monitored_vpcs.keys())}

if __name__ == "__main__":
    result = audit_vpc_flow_logs()
    print(json.dumps(result, indent=2))

Preventative Enforcement Policies

Log Archive S3 Bucket Policy for Multi-Account VPC Flow Logsjson
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowOrgVPCFlowLogsDelivery",
      "Effect": "Allow",
      "Principal": {
        "Service": "delivery.logs.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::central-vpc-flow-logs-LOG_ARCHIVE_ACCOUNT_ID/AWSLogs/*",
      "Condition": {
        "StringEquals": {
          "aws:SourceOrgID": "o-xxxxxxxxxx",
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    },
    {
      "Sid": "AllowVPCFlowLogsAclCheck",
      "Effect": "Allow",
      "Principal": {
        "Service": "delivery.logs.amazonaws.com"
      },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::central-vpc-flow-logs-LOG_ARCHIVE_ACCOUNT_ID",
      "Condition": {
        "StringEquals": {
          "aws:SourceOrgID": "o-xxxxxxxxxx"
        }
      }
    }
  ]
}
Terraform: VPC Flow Log with Parquet Format and Hive Partitionshcl
resource "aws_flow_log" "vpc_flow_logs" {
  log_destination      = "arn:aws:s3:::central-vpc-flow-logs-${var.log_archive_account_id}"
  log_destination_type = "s3"
  traffic_type         = "ALL"
  vpc_id               = var.vpc_id

  max_aggregation_interval = 60

  destination_options {
    file_classification_format = "parquet"
    hive_compatible_partitions  = true
    per_hour_partition          = true
  }

  tags = {
    Environment = "Production"
    ManagedBy   = "Terraform"
  }
}

Remediation Guide & Automation

Enable VPC Flow Logs on all existing and new VPCs delivering to the central Log Archive S3 bucket configured with Parquet file format, 1-minute aggregation interval, and Hive-compatible partitioning.

Step-by-Step Instructions

  1. Provision the central VPC flow logs S3 bucket in the Log Archive account with proper organization bucket policies.
  2. Configure VPC Flow Logs for all VPCs across all regions targeting the central S3 bucket destination.
  3. Set `traffic-type` to `ALL`, `max-aggregation-interval` to `60`, and `file-classification-format` to `parquet`.
  4. Deploy AWS Config rule `VPC_FLOW_LOGS_ENABLED` across member accounts with automated SSM remediation.
  5. Validate flow log delivery and verify partition structure in S3.
AWS CLI: Create VPC Flow Log with Parquet Destination Optionsbash
aws ec2 create-flow-logs \
  --resource-ids vpc-0123456789abcdef0 \
  --resource-type VPC \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::central-vpc-flow-logs-LOG_ARCHIVE_ACCOUNT_ID \
  --max-aggregation-interval 60 \
  --destination-options FileClassificationFormat=parquet,HiveCompatiblePartitions=true,PerHourPartition=true

We value your privacy

We use analytics cookies to understand how visitors interact with our site and to improve the user experience. You can choose to accept or decline these cookies.