Everything in the physical world is physics. (I’m just now realizing how related those words are…). But mathematics can also describe purely abstract concepts, which I don’t see how those are physics. It may be possible to make connections between some abstract concepts and our physical world, but it’s a strong and imo incorrect claim to make that that’s always true and that that means those abstract concepts are physics.