Centralized VPC Flow Logging with Partitioning and Compression
Specification & Description
Enable VPC Flow Logs across all Virtual Private Clouds (VPCs), subnets, and transit gateways, aggregating traffic telemetry into the central Log Archive S3 bucket formatted with Parquet compression and 1-minute aggregation intervals.
Technical Specifications
VPC Flow Logs capture network telemetry regarding IP traffic flowing to and from elastic network interfaces (ENIs) across all VPCs in the Landing Zone.
1. Multi-Account Network Telemetry Aggregation
- Flow logs from every workload, transit, and shared services account must be published directly to the dedicated Log Archive account S3 bucket using the
delivery.logs.amazonaws.comservice principal. - S3 bucket policies restrict delivery to authorized organization IDs via
aws:SourceOrgID.
2. Performance & Cost Optimization Best Practices
- Storage Format: Store flow logs in Apache Parquet format rather than plain text. Parquet provides columnar compression, reducing S3 storage costs by up to 80% and speeding up Amazon Athena / SIEM query execution.
- Hive-Compatible S3 Partitions: Enable Hive-compatible partitions (
/year=2026/month=08/day=21/) for optimized query pruning and data lifecycle management. - Aggregation Interval: Enforce a 1-minute (60-second) maximum aggregation interval on production VPCs to quickly capture ephemeral port scanning and fast lateral movement attempts.
3. Governance via AWS Config
- Automated AWS Config managed rules (
VPC_FLOW_LOGS_ENABLED) detect any newly provisioned VPC that lacks active flow logging and trigger automatic remediation.
Security Rationale
VPC Flow Logs provide the essential network forensic layer for detecting unauthorized lateral movement, port scans, command-and-control (C2) beaconing, and data exfiltration. Centralizing flow logs in an optimized Parquet format ensures consistent data retention, rapid threat investigation, and cost-effective SIEM ingestion across all multi-account workloads.
Business & Security Impact
Without centralized VPC Flow Logs, security teams are blind to network-level communication, preventing forensic reconstruction of network intrusions, DDoS attacks, or anomalous internal traffic.
Related Controls & Guardrails (3)
Compliance Framework Mappings (5)
Regulatory baselines and industry standards mapped to this control specification.
| Framework / Standard | Mapped Requirements & Controls | Controls |
|---|---|---|
CIS AWS Foundations Benchmark v3.0CIS AWS | 4.3 | 1 |
NIST 800-53 Rev 5NIST 800-53 | AU-12CA-7SC-7 | 3 |
ISO 27001:2022ISO 27001 | A.8.15A.8.20 | 2 |
SOC 2SOC 2 | CC7.2 | 1 |
PCI DSS v4.0PCI DSS | 10.2.11.2.4 | 2 |
* Compliance framework mappings are provided for informational architectural reference. Always verify control alignment against your organization's formal audit requirements.
Automated Detection & Rule Logic
Detection Engine: aws-api • Automated: Yes
Audits all VPCs across active regions to ensure that at least one active VPC Flow Log is enabled, traffic-type is set to ALL, and logs are delivered to the centralized S3 destination.
Preventative Enforcement Policies
Remediation Guide & Automation
Enable VPC Flow Logs on all existing and new VPCs delivering to the central Log Archive S3 bucket configured with Parquet file format, 1-minute aggregation interval, and Hive-compatible partitioning.
Step-by-Step Instructions
- Provision the central VPC flow logs S3 bucket in the Log Archive account with proper organization bucket policies.
- Configure VPC Flow Logs for all VPCs across all regions targeting the central S3 bucket destination.
- Set `traffic-type` to `ALL`, `max-aggregation-interval` to `60`, and `file-classification-format` to `parquet`.
- Deploy AWS Config rule `VPC_FLOW_LOGS_ENABLED` across member accounts with automated SSM remediation.
- Validate flow log delivery and verify partition structure in S3.