Submitted 11 months ago by Fermiverse@kbin.social to selfhosted@lemmy.world
https://kbin.social/m/selfhosted@lemmy.world/t/221341
I use some batch scripts in my proxmox installation. They are in cron.hourly and daily checking for virus and ram/CPU load of my LXC containers. An email is send on condition.
What are your tipps or solution without unnecessary load on disc io or CPU time. Lets keep it simple.
I’ve used monit for maybe 2 decades now. Works great and simple to use.
Nice, will take a look into it
github.com/awesome-foss/awesome-sysadmin#monitori…
I use netdata
I use zabbix to monitor everything, agent on each device uses around 30 mb of memory and with the Linux templates it can monitor just about everything on the server.
Does zabbix use a database continuously polling and storing data or is live data used for indication and/or triggers?
Zabbix stores all it’s data in a PostgreSQL or MySQL database. However… there are 2 ways that Zabbix Agents work. Either in passive mode, or in active mode.
Passive Agent = “poller” process on the Zabbix server sends a request to the agent asking for values for the items it’s monitoring (based on template applied to host). Depending on the number of hosts you’re monitoring and how many poller processes are configured to start with the Zabbix server, you may run into a situation where requests are queued because the poller process is too busy. Increasing the number of poller processes will fix this, but it also adds additional load to your DB as each poller process will connect to your DB to write data, and each poller process will consume a certain amount of memory. Too many and you’ll run out of RAM, or bog down your DB.
Active Agent = “trapper” process on the Zabbix server listens for item values from being sent to it from the agents. Agents will query the Zabbix server to see what templates are applied to it’s host, and will figure out what items it’s supposed to monitor. The agent will actively query the items without the Zabbix server requesting it, and will send the item values to the server as scheduled. This puts a lot less load on the Zabbix server.
Item values are not read from the DB to activate the trigger. When a value is received that matches the trigger’s expression, then the trigger is activated. Live values are used to activate triggers and trigger actions (alerts).
Not the above guy but I believe it’s a database.
I set up custom bash scripts collecting information (df, docker json, smartCTL etc)
Either parse existing json info or assemble json strings and push it to Homeassistant REST api (cron)
In Homeassistant data is turned into sensors and displayed. HA sends messages of sensors fail.
Info served in HA:
Trying to keep my servers as barebones as possible. Additional services/apps put strain on CPU/RAM etc. Found out most of data necessary for monitoring is either available (docker json, smartCTL json) or can be easily caught, e.g. ‘’’df -Pht ext4 | tail -n +2 | awk '{ print $1}’’’ It was fun learning and defining what must be monitored or not, and building a custom interface in HA.
Thats basically the way I do it.
pvesh get /cluster/resources --output-format json-pretty | jq --arg k "lxc/$container_id" -r 'map(select(.id == $k))[].name, map(select(.id == $k))[].mem, map(select(.id == $k))[].maxmem, map(select(.id == $k))[].cpu')
Example using pvesh in proxmox. The data is available, just have to use it. I also prefer barabone approach.
I used monitorix a long time ago. now netdata.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
Fewer Letters | More Letters ---|--- DNS | Domain Name Service/System SSD | Solid State Drive mass storage VPN | Virtual Private Network
[Thread #11 for this sub, first seen 19th Jul 2023, 17:40] [FAQ] [Full list] [Contact] [Source code]
Nagios. Core, but i’ve worked with it for years and am kinda masochistic. (Currently tying it into an IDRAC6)
I’m ever so slowly teaching myself Zabbix, need something full-featured because I also need monitoring for my hosting clients etc
Might be a bit more complex than what you want, but I love Prometheus + Alertmanager and a nice sexy Grafana dashboard
Monit for simple stuff and daemon restart on failure. LibreNMS for SNMP polling, graphing, logging, & alerting.
FrostyCaveman@lemm.ee 11 months ago
Prometheus, Loki and Grafana.
And so so many Prometheus metric exporters.
Observability is such an endless rabbit hole, it’s so easy for me to spend huge amounts of time accomplishing not that much lol. But very enjoyable and cool to see it all come together.
My pro tips: using Kubernetes actually makes this stuff a heck of a lot easier to set up thanks to the common patterns that k8s has - lots of turnkey helm charts out there that make it all so easy and are powerful. Another tip would be to use Prometheus service discovery if you can. Also, Loki/Promtail is actually quite easy to set up - but using LogQL queries can be very tricky. Just be warned, observability is a full time hobby in itself lol
kill_dash_nine@lemm.ee 11 months ago
I’m in a similar boat except I just do everything on standard Docker containers but so do use Telegraf, Influx, and Grafana for everything. I’ve gone mostly to Discord notifications on any alerts. If I run into any problem scenarios, I figure out how to monitor it and add it via Telegraf and add an alert. I’m still just using Grafana alerts but it works fine for my home lab.
Even better if I can automate fixes to those problems.