So, recently I spun up cAdvisor to provide some metrics for the Grafana dashboard. I created both the docker-compose.yml and prometheus.yml thusly:
prometheus.yml:
spoiler
scrape_configs: - job_name: cadvisor scrape_interval: 5s static_configs: - targets: - cadvisor:8080
docker-compose.yml
spoiler
services: prometheus: image: prom/prometheus:latest container_name: prometheus ports: - 9090:9090 command: - --config.file=/etc/prometheus/prometheus.yml volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro depends_on: - cadvisor cadvisor: image: gcr.io/cadvisor/cadvisor:latest container_name: cadvisor ports: - 8080:8080 volumes: - /:/rootfs:ro - /var/run:/var/run:rw - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro depends_on: - redis redis: image: redis:latest container_name: redis ports:
- 6379:6379
Placed them both in /tmp/cadvisor/ and ran docker compose up. All well and good, got some metrics to feed Grafana and all would seem jippity jippity.
Next day I notice Prometheus is off line. Hmm, check everything out. Logs complaining of a missing prometheus.yml. On a hunch I recreated the above prometheus.yml and placed it back in /tmp/cadvisor/, restart Prometheus, and it fires right up no runs, no drips, no errors. Before I uploaded the new prometheus.yml, I notice that there is a directory now named prometheus.yml in /tmp/cadvisor/, which is empty. Deleted it.
Next day, same scenario. Missing prometheus.yml, directory called prometheus.yml in /tmp/cadvisor/. I thought well, if it’s getting deleted, change the permissions, and continued my daily affairs.
Today, same exact scenario. So, wtf, over? Run some commands:
stat /tmp/cadvisor/prometheus.yml sudo lsof /tmp/cadvisor/prometheus.yml grep "delete" /var/log/syslog
I can see that the file IS being deleted, but I cannot seem to trace down what is deleting it. It’s like there is a cron job that fires off every day at a certain time and deletes prometheus.yml, and in it’s place, creates a directory called prometheus.yml effectively taking Prometheus offline. I have no such cron job tho.
Any ideas? Suggestions? Ancient wizardry? Any mystical incantations or tomes to consult?
cass80@programming.dev 2 days ago
You have two things happening at once.
First. There a process that cleans up tmp files according to a configuration. Yours is probably set to clean files older than a day.
www.freedesktop.org/software/…/tmpfiles.d.html for more information
Second, As to the folder. Docker will create a folder with the bind mount’s name if it’s not found. So Docker tries to find your promethius file, doesn’t find it, then creates a folder with that name and mounts it instead.
You should move the files out of tmp. That’ll solve all your problems.
irmadlad@lemmy.world 2 days ago
Done. Standing by to stand by.
Thank you!
speculate7383@lemmy.today 2 days ago
That’s like keeping your lunch laying outside on the sidewalk, getting stepped on by people and destroyed, and then wondering if your lunch would be safer if you put it in a stronger bag (but still left it on the sidewalk).
Don’t leave your lunch outside laying on the sidewalk, regardless of what you might do to “protect” it. Don’t keep important files in /tmp
cass80@programming.dev 2 days ago
My guess is the cleanup process is running as root and clobbers anything it sees regardless of permissions. But that’s a guess. I’ve never tried keeping long term data in tmp.