Submitted 8 months ago by Trincapinones@lemmy.world to selfhosted@lemmy.world
How do you monitor your server containers, disks, load…?
Do you use an easy-to-use web interface? Do you do everything via SSH? Or maybe you’ve got a more complicated setup?
I want to change my setup and I’m looking for new ideas, I’ve been using Cockpit for some years and some of the plugins are really outdated (ZFS for example) and others are completely broken (docker-compose).
My own server? YOLO
At work? Grafana, KOBS, Victoria Metrics, Jaeger, OpsGenie, …
My own server? YOLO
I can’t figure out whether there’s a monitoring tool called YOLO or you don’t monitor anything.
Now I am intrigued to develop one that is called YOLO.
But just in case: no, I don’t monitor my server. If I notice something not working, I ssh into the machine and check what’s up. I don’t want to deal with another zoo of services for the monitoring part.
Yes.
This is the first time I’ve heard of Victoria Metrics. It looks like it has a similar use case as Prometheus, is that correct? If so, what made you or your team choose one over the other?
IIRC it had better performance than Prometheus. We also ditched Elasticsearch in favor of ClickHouse to keep up with log ingestion.
“Huh weird, I tried to use <insert service here> and it’s not working. Welp, guess I better fix it…”
I’m a huge fan of Netdata, very configurable and monitors just about anything you could want. Great interface and alerts too - www.netdata.cloud
Same been running netdata for years. They’re monetizing now where it used to just be free. Good for them, it’s a great product. And it’s foss
I was looking for something free that I could host on my machine but thanks, I didn’t know about it
Netdata is free and can be run standalone. Just install it and do not configure the cloud integration. You can see your dashboard on localhost:19999
As others stated, you can run and access the interface directly for free. Cloud dashboard is also free for up to 5 nodes. They recently added a flat-rate “Homelab” plan as well, if you want to remove the limit. It’s all quite usable for $0 otherwise though!
Netdata 100%
It feeds my itch for more data than I know what to do with and it’s presented in one of the cleanest ways I’ve ever seen for so much info.
I love how easy to use NetData is, but when running it on my home servers it destroys their performance lol. Every once in awhile I check in to see if it runs better.
That’s strange, I’ve run it fine on some very underpowered hardware. Are you adding a specific monitoring integration with it, or just out of the box settings?
I just use homepage as my homepage :D
I can see simple CPU/RAM/storage stats and got widgets for almost all services, one of them is portainer so I can see if any service is stopped (most of them are running in docker). Also few services send notification on error or update
I know its not really a monitoring tool, but it works well enough for me
Node exporter on hosts, OpenTelemetry collector to scrape metrics and collect logs, shipping them to Prometheus and Loki, visualising with Grafana.
Try VictoriaMetrics. Basically the same feature set as Prometheus, but so much more resource friendly for homelab scale. I store some metrics for 12 months now, because it’s easy.
Do you have a name for the opentelemetry collector? I’m interested.
Use the Contrib version of the collector, it has many more receivers, processors and exporters
Similar setup here with additional exporters like cadvisor for container metrics and other components.
OpenTelemetry is awesome, but still a very fast moving project. Expect therefore more frequent updates and changes compared to more older and established projects.
Zabbix for agent / snmp based statistics.
Uptime Kuma for up/down states with a webhook notification into Discord so I get instant alerts on my phone when one goes down.
How has nobody in this thread said check_mk yet?
It’s free, you host it yourself. It’s built off of nagios, compatible with nagios plugins, supports snmp or agent based checks. It can email, SMS, slack or discord you when something breaks, you can write your own custom checks in any language that can output to a local console… I could never imagine even looking for something else.
+1 for check_mk.
It’s got a scriptable config file that begs for automation like mgmtConfig and it does SNMP. For me, that’s it. SNMP->mqtt->SNMP next year.
I started using Checkmk recently after it was mentioned here and I really like it. I’d used Zabbix a bit but was annoyed at how much work it took to get it to do what I want. Checkmk was a lot better right out of the box.
I like monit. It’s simple to setup and pretty flexible.
I used it as well until I found out I could just do it with systemd. baeldung.com/…/systemd-service-fail-notification
I’ve been using uptime Kuma recently and it’s great but works better outside of docker.
Inside I’d yet a lot of false down positives from I assume docker throttling the checks.
For light touch monitoring this is my approach too. I have one instance in my network, and another on fly.io for the VPSs (my most common outage is my home internet). To make it a tiny bit stronger, I wrote a Go endpoint that exposes the disk and memory usage of a server including with mem_okay and disk_okay keywords, and I have Kuma checking those.
I even have the two Kuma instances checking each other by making a status page and adding checks for each other’s ‘degraded’ state. I have ntfy set up on both so I get the Kuma change notifications on my iPhone. I love ntfy so much I donate to it.
For my VPSs, this is probably not enough, so I am considering the more complicated solutions (I’ve started wanting to know things like an influx of fali2ban bans etc.)
I just do web hosting for clients sites and use Kuma to monitor uptime and SSL certificates.
Ive got multiple Kuma’s running as well.
Grafana set up to run on the server locally, then I connect to it via SSH forwarding.
I liked Grafana a lot, but I can’t monitor things like zfs pools with it right?
I don’t know as I don’t use zfs pools, but a simple search led me to this grafana.com/grafana/…/15362-zfs-pool-metrics/
Um, Proxmox?
I am running everything in docker compose so I’ve never found a use to it that justifies the waste of power
docker-compose doesn’t scale well and if you run it natively it is a little less secure.
Virtualization adds 1-2% of overhead at most and gives you way more control of how the hardware is used.
At home, nagios, at work colleagues. (I finally escaped the admin rat race)
At home, libreNMS. Just SNMP everything.
For work, whatever the tool of the day is from management.
Zabbix
Second Zabbix. Been using it for years and it just works.
Adding my vote for Zabbix. It was a bit of a bear to set up and I had to write custom scripts to install the agents with TLS settings that were secure enough for me, but once it’s all set up it’s amazingly easy and intuitive to use and incredibly customizable.
Grafana and Alertmanager. For graphing and sending webhooks into Discord and Email for me to see in real-time.
Prometheus and its exporters. If it has an exporter I can monitor it in Grafana. If it doesn’t have an exporter, I might be able to make one for it, since it’s just a plaintext static webpage with serialized metrics on it, anything can output that even PHP.
Infping/Influxdb for smokeping-like data in Grafana, works great for telling me about any packet loss or increased latency on any network that responds to ICMP echo. My uncle uses my services to complain to Spectrum about his Coax connection, which has led to him being able to have his ISP physically come out and stabilize it, now it hardly ever blips.
Also all of this is federated so if one server goes down I don’t lose 100% of my monitoring, and I can customize my data retention time based on how much disk I have locally. I use Ansible to automate updates and configuration.
It costs me 2-3 small-to-medium sized VPSes a month, or under $30/mo USD, which is a bargon compared to enterprise cloud solutions/services you can pay for that have small data retention or other limits. New Relic as an example is expensive if you want anything more than 30 days of data and they don’t offer as much as I get with my own monitors.
anything can output that even PHP.
That sounded pretty bitter.
Prometheus and Altertmanager
Monitorix or Netdata.
I use Proxmox, so I just use the PVE web interface
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
| Fewer Letters | More Letters |
|---|---|
| SSH | Secure Shell for remote terminal access |
| SSL | Secure Sockets Layer, for transparent encryption |
| TLS | Transport Layer Security, supersedes SSL |
[Thread #521 for this sub, first seen 17th Feb 2024, 17:25] [FAQ] [Full list] [Contact] [Source code]
Htop and logwatch
i just have top running through ssh on an xterm window.
For the physical hosts / bare metal I use fluentbit, with Loki as the backend. Grafana for visualization and alerts. This gives me utilization metrics and uptime monitoring. The app containers themselves I do not monitor.
Prometheus + node/container exporters. + Grafana for dashboards I haven’t touched zabbix in years but last time it didn’t support very well dynamic scalations. Also all of them are focused on monitoring infrastructure, you need to pay if you want APM or UX.
Enterprise level, for APM I like datadog, much better than NR. For UX we use acoustic tealeef.
Grafana, fronting information from Prometheus, Loki and Telegraf/influxdb since I’m used to that from work and has been a bit more set and forget compared to node_exporter. Easier to add in plugins as well instead of a new container/service to scrape.
Cockpit, Cosmos Cloud, Grafana, and a few other things. It’s not the most optimal solution.
I use collectd and graphs on my openwrt router. It can even use data from mqtt-connected thermometers and gather metrics from other collectd instances.
systemd can be used to run whatever notification scheme you would like to use whenever some service fails. Here’s an example of how to do it: baeldung.com/…/systemd-service-fail-notification
New Lemmy Post: What's Your Preferred Server Monitoring Method? (https://lemmy.world/post/12068316)
Tagging: #SelfHosted
(Replying in the OP of this thread (NOT THIS BOT!) will appear as a comment in the lemmy discussion.)
I am a FOSS bot. Check my README: https://github.com/db0/lemmy-tagginator/blob/main/README.md
Toes@ani.social 8 months ago
My clients when they text me the server is down.
fatboy93@lemm.ee 8 months ago
This has the same energy as my spouse yelling at me because jellyfin went down
Passerby6497@lemmy.world 8 months ago
Or my partners greeting me in the morning “Home assistant went down again, so the lights are all manual”
Thankfully that one is mostly solved.
TCB13@lemmy.world 8 months ago
So damn accurate ahhaha