Or can matrix only talk with matrix, IRC with IRC, XMPP with XMPP, etc?

Yes, they have to be the same protocol to communicate.