Selective CloudWatch Logs Aggregation with Cost Guardrails
Specification & Description
Selectively aggregate security-relevant, compliance-mandated, and audit-grade CloudWatch Log Groups from workload accounts into the central Log Archive account using CloudWatch Logs cross-account destinations, Amazon Data Firehose, and durable S3 cold storage. Explicitly filter out high-volume ephemeral debug, trace, and synthetic health-check streams to prevent severe CloudWatch ingestion and data transfer cost escalation.
Technical Specifications
While AWS CloudTrail captures management and data API events, workload environments (EC2, ECS, EKS, Lambda) generate critical application audit trails, authentication events, and operating system security logs within CloudWatch Logs. However, streaming all application logs to a central account is a costly anti-pattern that creates signal fatigue and runaway bills.
1. Selective Log Group Scope & Security Classification
- In-Scope for Centralization: Authentication/authorization events, administrative audit logs, financial transaction records, operating system audit streams (
/var/log/secure,/var/log/audit/audit.log), and production container error streams. - Out-of-Scope (Local Only): High-frequency debug/trace logs, ephemeral stdout container noise, synthetic ALB/NLB health-check probes, and non-production sandbox streams. These should remain in local accounts with short retention windows.
2. Cost Dimensions & Streaming Economics
- Cross-account log aggregation incurs multiple distinct cost layers:
- CloudWatch Logs Ingestion: ~$0.50 per GB ingested in member accounts.
- Cross-Account Data Transfer: Inter-account and cross-region egress network fees.
- Amazon Data Firehose Processing: ~$0.029 per GB ingested and delivered.
- S3 Storage & PUT Requests: Per-request and ongoing storage fees in the central Log Archive bucket.
- Subscription Filter Patterns: Enforce strict
filter_patterncriteria (e.g.,[time, level = ERROR || level = WARN || level = AUDIT, ...]) on subscription filters to drop benign debug lines at the source before transmission.
3. Retention Hygiene & Storage Optimization
- Member account CloudWatch Log Groups must enforce strict local retention periods (e.g., 7 to 14 days) to eliminate local storage fees ($0.03/GB/month).
- Long-term compliance retention is offloaded entirely to S3 Lifecycle policies (transitioning from S3 Standard to S3 Glacier Instant Retrieval / Deep Archive) in the central Log Archive account.
Security Rationale
Decentralized logging scattered across individual AWS accounts creates security blind spots during incident response. However, indiscriminately streaming all workload and debug logs across an enterprise leads to runaway CloudWatch and Data Firehose ingestion costs. Applying selective subscription filters balances forensic visibility and SIEM correlation with strict cost governance.
Business & Security Impact
Failing to centralize security logs delays incident response and risks permanent evidence loss if accounts are decommissioned. Conversely, failing to filter out high-volume debug and health-check logs results in massive, unbudgeted CloudWatch ingestion, data transfer, and Firehose processing charges.
Related Controls & Guardrails (4)
Compliance Framework Mappings (4)
Regulatory baselines and industry standards mapped to this control specification.
| Framework / Standard | Mapped Requirements & Controls | Controls |
|---|---|---|
NIST 800-53 Rev 5NIST 800-53 | AU-2AU-4AU-12 | 3 |
ISO 27001:2022ISO 27001 | A.8.15 | 1 |
SOC 2SOC 2 | CC7.2 | 1 |
PCI DSS v4.0PCI DSS | 10.2.1 | 1 |
* Compliance framework mappings are provided for informational architectural reference. Always verify control alignment against your organization's formal audit requirements.
Automated Detection & Rule Logic
Detection Engine: aws-api • Automated: Yes
Audits member account CloudWatch Log Groups matching security and production audit patterns (e.g., /aws/containerinsights/production/, /var/log/secure, /app/audit/) to verify they have an active cross-account subscription filter and short local retention (<= 30 days) configured.
Preventative Enforcement Policies
Remediation Guide & Automation
Provision a centralized CloudWatch Logs Destination in the Log Archive account with Amazon Data Firehose, and attach selective cross-account subscription filters with filter patterns to security-critical workload log groups.
Step-by-Step Instructions
- Classify application and system log groups to identify security-critical audit and error streams.
- Provision the central S3 log bucket (with Glacier lifecycle transitions) and Amazon Data Firehose stream in the Log Archive account.
- Create the CloudWatch Logs Destination with a resource policy allowing `aws:PrincipalOrgID` access.
- Deploy subscription filters with specific `filter_pattern` rules targeting only audit and error events to avoid streaming debug noise.
- Set explicit retention periods (7 to 14 days) on member account log groups to eliminate local storage cost accumulation.