Debugging Production With CloudWatch (Without SSHing Anywhere)

Containers changed how I debug. No SSH, no tailing files on the box, the box might not even exist for more than an hour. Everything has to go to a central place, and on AWS that place is CloudWatch — which I ignored for years in favor of fancier tools, then got humbled by.

Here's the workflow I use now when something is wrong and CloudWatch is all I have.

First: make sure the logs are actually there

This sounds too obvious to mention, and yet. Every ECS task, Lambda, and EC2 agent should ship to its own log group with retention set — forever-retention log groups fill up and cost more than the service they're debugging:

aws logs put-retention-policy \
  --log-group-name /app/api \
  --retention-in-days 30

Structured logs make everything downstream easier. One JSON line per event beats a wall of printf text when you need to filter. Even {"level":"error","user_id":42,"route":"/checkout"} with log keys that stay stable — inconsistent key names are how queries silently return nothing during an incident.

Logs Insights is the real tool

The old CloudWatch "view events" tab is for browsing. Logs Insights is for answering questions. A query I've run a hundred times — errors in the last 15 minutes, grouped by route:

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

With structured logs it gets sharper:

fields @timestamp, status, route, request_id
| filter status >= 500
| stats count() by route, bin(5m)
| sort count_ desc

That second one shows which route is failing and when it started — often the two facts you need before anyone asks "is it still happening?"

For tracing a single broken request, grab the request ID from the client error and:

fields @timestamp, @message
| filter request_id = "abc-123"
| sort @timestamp asc

Now you have the full journey of that one request across services, assuming you propagated the ID. If you didn't propagate it — add that to the backlog before the next incident, not during it.

Save your queries. Logs Insights lets you bookmark them; I keep a small set named 5xx-by-route, slow-requests, auth-failures. During an incident is the worst time to write PPL from scratch.

When you need a number, not text: metric filters

Logs answer "what happened." Metrics answer "how much, since when." A metric filter turns log lines into CloudWatch metrics you can graph and alarm on:

aws logs put-metric-filter \
  --log-group-name /app/api \
  --filter-name five-hundred-errors \
  --filter-pattern '?ERROR "?status=5"' \
  --metric-transformations \
    metricName=FiveHundreds,metricNamespace=App,metricValue=1,defaultValue=0

Now alarm on Sum >= 10 for 2 minutes and get paged — or better, get a dashboard widget that makes the problem visible without anyone paging.

Alarms I actually keep

The trick to surviving on-call is fewer, sharper alarms:

• Error rate, not raw error count — one bad client retrying shouldn't page you.
• Latency p95 over SLO, with a burn-rate window, not "p99 spiked once."
• Queue depth / saturation before users notice — saturation leads, symptoms follow.
• Synthetic canary on the checkout path — the only alarm that fires when nothing looks wrong in your logs because the feature is silently broken.

Everything else goes to a dashboard. If an alarm doesn't demand a human at that hour, it's a dashboard widget with extra steps.

Composite view for the actual incident

My order of operations when the pager goes off:

1. Alarm → which service, which metric, when it started.
2. Dashboard → single-number health: error rate, latency, saturation, deploy marker.
3. Logs Insights → slice by route/status in the start window.
4. Correlate with recent events — a deploy, a scaling action, a third-party status page — the deploy marker on the dashboard exists precisely for this moment.

SSH would have given me a shell on one box and a false sense of thoroughness. CloudWatch gives me every box at once. I don't miss the shell.