Yeah unfortunately you’re depending on proper behavior from other people’s systems, which is almost never a safe thing to do.

I’m sure there’s some magic you can do to only have one ns backed by two servers, but the easiest fix would be to randomize the order that the ns records are presented. It would reduce, but not eliminate, the failure you describe. If ns1 is down but ns2 is listed first, even a poorly-behaved resolver should be able to resolve your stuff.