Selective CloudWatch Logs Aggregation with Cost Guardrails

Selectively aggregate security-relevant, compliance-mandated, and audit-grade CloudWatch Log Groups from workload accounts into the central Log Archive account using CloudWatch Logs cross-account destinations, Amazon Data Firehose, and durable S3 cold storage. Explicitly filter out high-volume ephemeral debug, trace, and synthetic health-check streams to prevent severe CloudWatch ingestion and data transfer cost escalation.

While AWS CloudTrail captures management and data API events, workload environments (EC2, ECS, EKS, Lambda) generate critical application audit trails, authentication events, and operating system security logs within CloudWatch Logs. However, streaming all application logs to a central account is a costly anti-pattern that creates signal fatigue and runaway bills.

1. Selective Log Group Scope & Security Classification

• In-Scope for Centralization: Authentication/authorization events, administrative audit logs, financial transaction records, operating system audit streams (/var/log/secure, /var/log/audit/audit.log), and production container error streams.
• Out-of-Scope (Local Only): High-frequency debug/trace logs, ephemeral stdout container noise, synthetic ALB/NLB health-check probes, and non-production sandbox streams. These should remain in local accounts with short retention windows.

2. Cost Dimensions & Streaming Economics

• Cross-account log aggregation incurs multiple distinct cost layers:
  • CloudWatch Logs Ingestion: ~$0.50 per GB ingested in member accounts.
  • Cross-Account Data Transfer: Inter-account and cross-region egress network fees.
  • Amazon Data Firehose Processing: ~$0.029 per GB ingested and delivered.
  • S3 Storage & PUT Requests: Per-request and ongoing storage fees in the central Log Archive bucket.
• Subscription Filter Patterns: Enforce strict filter_pattern criteria (e.g., [time, level = ERROR || level = WARN || level = AUDIT, ...]) on subscription filters to drop benign debug lines at the source before transmission.

3. Retention Hygiene & Storage Optimization

• Member account CloudWatch Log Groups must enforce strict local retention periods (e.g., 7 to 14 days) to eliminate local storage fees ($0.03/GB/month).
• Long-term compliance retention is offloaded entirely to S3 Lifecycle policies (transitioning from S3 Standard to S3 Glacier Instant Retrieval / Deep Archive) in the central Log Archive account.