As someone who came into the series at elden ring, I agree with you. I initially tried a mage and it felt quite a bit more difficult than sword and board.

The game felt like it was designed for physical weapons and shield, whereas with magic the difficulty was all over the place - easy time with one boss, nearly impossible on another. If you aim for a boss cheese build maybe magic is easier.