It is more complicated than that. Vulnerable minorities don’t thrive in spaces where they’re endlessly playing whack a mole with bigots and trolls.
I didn’t leave Twitter for the fediverse because of its network. I left even before Musk, because Twitter was full of hate that the admins didn’t feel the need to action.
Your preference is just that… And as long as there is room for your preference on the fediverse without it being the only way to experience it, we can both have the experiences we want.