Centralized VPC Flow Logging with Partitioning and Compression
Specification & Description
Enable VPC Flow Logs across all Virtual Private Clouds (VPCs), subnets, and transit gateways, aggregating traffic telemetry into the central Log Archive S3 bucket formatted with Parquet compression and 1-minute aggregation intervals.
Technical Specifications
VPC Flow Logs capture network telemetry regarding IP traffic flowing to and from elastic network interfaces (ENIs) across all VPCs in the Landing Zone.
1. Multi-Account Network Telemetry Aggregation
- Flow logs from every workload, transit, and shared services account must be published directly to the dedicated Log Archive account S3 bucket using the
delivery.logs.amazonaws.comservice principal. - S3 bucket policies restrict delivery to authorized organization IDs via
aws:SourceOrgID.
2. Performance & Cost Optimization Best Practices
- Storage Format: Store flow logs in Apache Parquet format rather than plain text. Parquet provides columnar compression, reducing S3 storage costs by up to 80% and speeding up Amazon Athena / SIEM query execution.
- Hive-Compatible S3 Partitions: Enable Hive-compatible partitions (
/year=2026/month=08/day=21/) for optimized query pruning and data lifecycle management. - Aggregation Interval: Enforce a 1-minute (60-second) maximum aggregation interval on production VPCs to quickly capture ephemeral port scanning and fast lateral movement attempts.
3. Governance via AWS Config
- Automated AWS Config managed rules (
VPC_FLOW_LOGS_ENABLED) detect any newly provisioned VPC that lacks active flow logging and trigger automatic remediation.
Security Rationale
VPC Flow Logs provide the essential network forensic layer for detecting unauthorized lateral movement, port scans, command-and-control (C2) beaconing, and data exfiltration. Centralizing flow logs in an optimized Parquet format ensures consistent data retention, rapid threat investigation, and cost-effective SIEM ingestion across all multi-account workloads.