Comment on Hiw do you observe your server funktions?
FrostyCaveman@lemm.ee 11 months ago
Prometheus, Loki and Grafana.
And so so many Prometheus metric exporters.
Observability is such an endless rabbit hole, it’s so easy for me to spend huge amounts of time accomplishing not that much lol. But very enjoyable and cool to see it all come together.
My pro tips: using Kubernetes actually makes this stuff a heck of a lot easier to set up thanks to the common patterns that k8s has - lots of turnkey helm charts out there that make it all so easy and are powerful. Another tip would be to use Prometheus service discovery if you can. Also, Loki/Promtail is actually quite easy to set up - but using LogQL queries can be very tricky. Just be warned, observability is a full time hobby in itself lol
kill_dash_nine@lemm.ee 11 months ago
I’m in a similar boat except I just do everything on standard Docker containers but so do use Telegraf, Influx, and Grafana for everything. I’ve gone mostly to Discord notifications on any alerts. If I run into any problem scenarios, I figure out how to monitor it and add it via Telegraf and add an alert. I’m still just using Grafana alerts but it works fine for my home lab.
Even better if I can automate fixes to those problems.