The only reason Musk has influence is because he has an audience on Twitter. The only reason he has an audience is because people are willing to listen to what he has to say.

The questions is: who the fuck is interested in what Elon Musk has to say?

Unfollow him - or whatever ignoring someone on Twitter entails, I’ve never been on Twitter so I don’t know - and Musk will find himself deprived of influence. Just like that.