tenchiken
@tenchiken@anarchist.nexus
- Comment on What is your weirdest self-hosting problem you've had to solve? For me: Mouse inside the server. 3 days ago:
Power isn’t so bad really. These were all 2tb disks, so ended up with about 40tb total after shenanigans.
Cabling was atrocious. I’d originally been using several pulled 4 disk SAS backplanes from servers thrown out of my work, just screwed down to some pine boards in a stack.
Zip ties and sata/sas cables were the rule.
Overall the drives had 4 small power supplies initially, 300watt iirc. Cheap.
Somewhere I have an old photo from the wiring during the “rescue” but I have to look around.
Disk failure was actually not an issue for ages before. A specific thunderstorm with 2 nearby tornadoes is what caught me this time. Before that, the occasional single disk every 6 months or so.
I still have running arrays using 15 year old disks, many with more than 10 years of that powered on. If you find good models, they can last ages.
- Comment on What is your weirdest self-hosting problem you've had to solve? For me: Mouse inside the server. 3 days ago:
Mine would be when I had a 48 bay disk array / JBOD fail on me… badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.
These were all running bcache on top of mdraid… One single mount.
Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.
I just HAD to solve this one though.
In come several spare chassis / mobo etc…. get a bunch of drives powered and on /dev/ , move to the next.
A couple spare gigabit switches…
several gigabit NICs…
two explicit paths for each machine…
a bit of iSCSI magic, and one machine now had the physical disks all exposed to it… mdadm –assemble blah blah, bit of UUID chaos…
It’s surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.
Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of “I can lose this and don’t care” vs “this box gets RAID10 and/or offsite backups nightly”.
back up your critical stuff people!
- Comment on SMART data on some old drives, can anyone beat my powered-on times? 3 weeks ago:
root@kamidake:~# for i in /dev/sd? ; do echo $i; smartctl -a $i |grep Hours| awk ' { print $10/8760}’; done
/dev/sda
15.6055
/dev/sdb
15.6979
/dev/sdc
6.04909
/dev/sdd
7.35708
/dev/sde
10.618
/dev/sdf
7.28071
/dev/sdg
10.5689
/dev/sdh
10.6264
/dev/sdi
2.28311
/dev/sdj
10.7498
/dev/sdk
10.5832
/dev/sdl
7.35502
/dev/sdm
10.643
/dev/sdn
10.8016
/dev/sdo
10.7558
/dev/sdp
9.11416
root@kamidake:~#
These are running ZFS and I have 2 other boxes with similar, most in the 10-12 year range. Spinning drives.
Good stuff.