Cloud Monitoring & Observability Platform

Terraform, ECS Fargate, cross-account CloudWatch

A Terraform-provisioned observability stack on ECS Fargate — Grafana and uptime monitoring behind an internal-only HTTPS load balancer, reading metrics and logs across a three-account AWS organization through assumed roles.

TerraformECS FargateGrafanaCloudWatchFireLensRoute53GitLab CIIAM

## Problem

Metrics, logs, and uptime signals lived in whichever account happened to own the workload. When the application moved to a dedicated production account, every dashboard pointing at the old account silently went stale — the panels still rendered, they were just describing nothing.

## Architecture

Grafana and an uptime monitor run as ECS Fargate services behind an internal-only HTTPS ALB, with ACM certificates resolved through a private Route53 zone. Credentials live in Secrets Manager, dashboards and datasources are provisioned as code at deploy time, and the monitoring database is restored from S3 by a sidecar on every task start. GitLab CI stages the whole stack precommit, DAST, plan, then a manual apply gate.

## Cross-account access

The monitoring account's task role assumes a read-only role in the production account for CloudWatch metrics and Logs Insights. One detail cost real time: the logs policy must scope its Resource to the log-group ARN pattern rather than a bare wildcard, or cross-account StartQuery calls fail with AccessDenied while metric panels keep working — so the dashboard half-works and looks like a Grafana bug rather than an IAM one.

## Storage: the failure worth knowing

The uptime monitor hardcodes SQLite WAL journal mode, and WAL is fundamentally incompatible with NFS-backed storage — mounting its data directory on EFS produces SQLITE_CORRUPT on startup. The fix was block storage plus S3 restore, and the database must be converted out of WAL mode before upload or the restored copy crashes the same way. Shared filesystems are not a drop-in substitute for block devices when the application assumes POSIX locking.

## Log routing

Container logging moved from the default driver to FireLens with FluentBit, classifying web-server output by HTTP-method and error-pattern regex and re-emitting each class to its own log group while the originals keep flowing to the aggregate stream.

## Lessons Learned

A CI script without `set -e` silently swallowed image build failures and exited zero, so the pipeline reported green while the registry kept serving the previous image — every deploy 'succeeded' and nothing changed. Green is not the same as correct; a build step has to fail loudly. Separately, a Fargate-managed network interface survived a destroy and blocked its security group's deletion. Because the interface is requester-managed, the account owner cannot remove it at all — recognising an unfixable resource early is worth more than an afternoon of trying.

// ls ../

FLAGSHIP

Cloud Monitoring & Observability Platform

## Problem

## Architecture

## Cross-account access

## Storage: the failure worth knowing

## Log routing

## Lessons Learned

Snock →

SpecterHeal →

Enterprise DDoS Defense →

Hardened GitLab DevSecOps Platform →

Trend Micro XDR Remediation →

AWS Incident Response Automation →

SOC Automation & Visualization →

Fortinet SD-WAN Deployment →

CTF Training Platform →

CIS Benchmark & Remediation Program →

Open Banking Auth Flow Load Testing →