Datadog is a good product. I remember the first time I used it, it blew my mind having ALL OF THAT VISIBILITY into everything. All the little premade integrations as well, it slotted straight into our app and infrastructure and provided visibility I had never even dreamed of.

The problem is pricing. It is unjustifiably expensive for random little side projects, and it is exorbitantly priced for bigger companies, and it gets more and more expensive the more you integrate with Datadog. Logs, APM, new machines, etc, it all costs -a lot-. Every time one of my teams wanted to instrument more of our systems, we had to think about its impact on business costs, and it was a whole thing.

And for what? A database that stores logs, and shows them with a pretty UI? Like I get it if you're petabyte scale and need some SERIOUS web-scale observability. But a company doing a few GBs of data / month (even a few TBs)? You can literally buy 32TB disks for ~$800, and you get your 30/60/90-day retention forever.

So I rebuilt the whole thing on my own side-project servers, as a proof-of-concept. Prometheus for metrics, Loki for logs, Tempo for traces, Grafana as the neat UI, and OpenTelemetry to standardise the way I emit telemetry in my projects (no ddtrace agent to install!).

My apps don't need a backend in their config at all; they emit OTLP into a collector, and the collector will route it to wherever I've pointed it.

The traces are the part that makes me super happy, because they get me thinking about how nice it'll be when I hook this up to real projects, with the whole web > backend > db distributed tracing thing. I had that in Datadog, but it was expensive and I always felt sad that I couldn't expand my use of it. But now, it's all free:

Obligatory AI mention: Thanks to AI, I set the whole thing up with Ansible and learned a ton along the way and the "learning curve" of self-hosting your own observability went down dramatically. I had had Grafana in my "to figure out" list for the longest time and now... well, it's figured out.

All of it is open source and runs on hardware I'm already paying for, so the only real cost of the stack is disk space. This isn't the same as unlimited, but my limit is however much disk space I'm willing to buy, and disk space is cheap, so who cares. I keep everything, for as long as I feel like keeping it, and I never ever have to think about how much it'll cost me to hook up a new project or a new trace or a new event or a new -anything- to observability. I now have it for free forever. It's LIBERATING.

I haven't even gotten into Faro / RUM yet, that's next! Another thing I get for free with my setup that'd cost more money in Datadog.

The tradeoff is that I'm now responsible for my own monitoring. If the Grafana box falls over, I have a dead man's switch in healthchecks.io, and that'll notify me so I can deal with it. That's fine. An observability outage isn't a big deal for my particular needs; 30 minutes of not having logs stored is not gonna kill anyone, there's no real business cost to it. Obviously we want to avoid it, but it's a risk I'm willing to live with. If I were running this for a company at a scale where an outage would cost real money, I might weigh it differently. But for everything I run, this is awesome.

And, yeah, yeah, "what you're paying for is someone to administer this for you and deal with all the difficulties of it". But I have the server resources already, and I have the skills in-house (lol myself), so why the heck would I outsource this? It's a one-off setup cost, time-wise, and maybe there'll be a little bit of maintenance once in a blue moon. And that's... it. That's not worth hundreds of dollars per month. It just isn't. The money I save every month on Datadog is money that can go to pay for things that would genuinely be difficult for me to do, or that would require recurring effort to maintain monthly. Leaving an Ansible-managed service running isn't one of those.

Anyway. I'm just excited. I finally have Datadog-like observability in my side projects, FOR FREE. Woohoo!