Here is exactly what happened. We run an old Node 14 box for image processing crons. Nobody touches it because the dev who wrote it left three years ago and the webpack setup makes no sense. Our junior dev Rahul just wanted to track execution time, so he pushed a quick commit on Friday:

statsd.histogram('image.process_time', duration, tags: ['uuid:' + file.id])

He passed the file UUID directly as a metric tag. Datadog doesn't care if it's a mistake. They treat every single unique tag value as an entirely separate custom metric. The cron chewed through 300,000 images over the weekend, and Datadog's pricing model happily registered every single one.

This is the state of observability right now. You either get ripped off by SaaS vendors or you fight with AWS tooling.

AWS Native Tools Are a Trap

CloudWatch's 5-minute default aggregation is a joke. We once had a payment webhook worker with a memory leak—it would crash and systemd would bounce it back up inside 45 seconds. Webhooks were failing, but the CloudWatch EC2 dashboard showed a flat green line. AWS smoothed out that 45-second dip over a 5-minute bucket, so the incident never showed up on the graph. Want 1-minute resolution? Enter your credit card for "Detailed Monitoring."

Then comes CloudWatch Logs Insights. Querying cross-service logs during an incident is pure pain. You are in the middle of a triage, typing fields @timestamp, @message | filter @message like /Timeout/ | sort @timestamp desc. You wait 40 seconds, the query times out, you narrow the window to 15 minutes, re-run, and then AWS logs you out because the SSO token expired. When you finally get the output, the payload is clipped because nobody changed the Docker log driver limits.

We bought Datadog to stop dealing with that garbage.

Why Datadog is Brilliant (And a Financial Nightmare)

Their APM works. You drop the agent, inject headers, and immediately see what Postgres query is stalling your Node event loop. We had a cron running an unbatched UPDATE across our users table that locked the whole thing, queued API traffic, and threw 504s via Nginx. In Datadog, that exact query was highlighted in red. Took two minutes to find it, kill the cron, and slack the team. That visibility is addictive, and once you get used to it, plain log files feel impossible.

The tradeoff is monitoring your Datadog bill like a second job. We spend more engineering hours writing Exclusion Filters to kill INFO logs at the agent level than building product features. Send a simple "User logged in" event at scale and you'll blow your monthly budget in a week. Drop logs before they leave your VPC or prepare to get yelled at by finance.

Prometheus Isn't Actually Free

People think self-hosting Prometheus is the answer to the billing problem. It isn't. I set up Prometheus cluster for a Go microservices stack two years ago. The base setup was fine, but the pod kept getting OOMKilled every few hours. A developer had added an 'HTTP User Agent' label to a request metric. High cardinality takes down Prometheus local memory just like it spikes bills in Datadog. Spent nearly a week tweaking retention flags and memory limits just to stop the crash loop.

PromQL is unreadable at 3 AM when you need to calculate an error budget. Local storage eats an EBS volume within weeks. Want long-term retention? Now you have to maintain Thanos or Cortex alongside your actual product infrastructure.

And Grafana setups always decay. Our old DevOps guy Amit left years back, and his "Prod-Main-Final-FINAL" dashboard is still the only place showing accurate Redis cache hit rates. We can't update it because he hardcoded server IPs everywhere. Out of 60 dashboards, maybe five actually work.

Stop Clicking in the AWS Console

One quick fix if you're wiring up alarms right now: stop clicking around the AWS console. If you attach a CPU alarm to an EC2 instance via the UI, an Auto Scaling Group will replace that instance tomorrow, give it a new ID, and silently drop your alert.

Use Terraform:

resource "cloudwatch_metricalarm" "cpu_high" {
alarm_name = "${var.service_name}-cpu"
comparison_operator = "GreaterThanThreshold"
evaluation_time = "3"
metric_name = "CPUUtilization"
namespace = "AWS/EC2"
period = "60"
statistic = "Average"
threshold = "85"
alarm_actions = [slack_notification.alerts.arn]
}

Keep evaluation_time at 3. At 1, every Node GC pause triggers a false alarm in your incident channel.

The Ugly Truth About Alert Fatigue

Most teams don't understand the difference between monitoring a metric and alerting on it. Your dashboard should have everything. It should show CPU, disk I/O, memory, and network traffic so that when you are actively debugging an incident, you have the historical context to figure out what broke.

But you should absolutely never attach a PagerDuty rule to 95% of those graphs. An alert means a human needs to wake up at 3 AM and fix something immediately. A background worker pinned at 90% CPU doesn't matter if the checkout button still responds in under 200ms. CPU alerts just lead to alarm fatigue. Our lead engineer got so sick of being woken up at 4 AM by a batch job spiking a background worker that he literally wired PagerDuty into a Zapier webhook to auto-resolve any ticket with "High CPU" in the title. When your engineers are actively building automation to ignore your monitoring tools, your alerting strategy is broken.

If you stop alerting on CPU, what's actually left? The stuff that breaks the user experience.

I page on exactly four things. The p95 API latency, because "average latency" is a complete lie. Your average latency might look like a healthy 40ms because 95% of your traffic is just hitting a cached health check endpoint. Meanwhile, the actual checkout API is taking 12 seconds because someone forget an index on the carts table. If you alert on the average, you will literally never know the site is broken until customers start complaining on Twitter.

I also alert on unhandled HTTP 500s. The Redis queue depth, because if the backlog is just growing, the workers are dead. And the database connection pool. That's literally it.

Everything else—like disk usage slowly creeping up to 80%—is just a monitoring metric. You put it on a dashboard and look at it on Monday morning.

So, What Should You Actually Use?

People always ask what they should use.

Honestly? If you have the money, pay Datadog. The APM will save you hours of debugging. Just assign a developer to babysit the billing page so you don't go broke.

If management refuses to pay for SaaS, you're stuck running Prometheus. Just don't trick yourself into thinking it's free. The software is free, but you'll end up paying a platform engineer a full-time salary just to keep the Thanos storage cluster from falling over.

And if you just have a couple of basic AWS servers, literally just use CloudWatch until you outgrow it. Don't overengineer it.

Whatever you do, drop your debug logs at the edge, alert on the actual errors, and for the love of god, keep UUIDs out of your metric tags.