As a business owner, I can even predict the immediate outcome of rejecting OP. They would come to me, and say “Why did they get the job? I thought it was mine.”

“Well, you indicated that if you took the job, you wouldn’t be able to work with members of the team, so we chose someone who could. Maybe you’ll be a better fit for the next one.” And then they’d never be offered another promotion.

Ultimatums like “It’s them or me,” seldom work out well for the issuer.