If Lemmy.World went away, then correct you would not "lose" the users as, well you said it, they would simply move to another instance.

But if Lemmy.World remained and you blocked it (if you had a method to do that - it's not easy at all using base Lemmy but it is doable with some older apps or like Ublock Origin filter rules and such), then in that context you would indeed "lose" all of that content. Or like if you got banned from that instance then that's another way that you could "lose" access to engage with communities located on it.

The more centralized something is, like Reddit, the more damaging it is to lose access to it, while the more decentralized, as you pointed out, the less overall effect that perturbations have upon the network.