“I’m still friends with his brother, so it’s impossible for me to completely block him out from my life unfortunately.”

No, it is possible. If you happen to lose his brother as a friend then that’s an unfortunate consequence.

Life’s too short to have to deal with assholes like that. Not worth your limited time on this planet. Find new friends.