when it clearly should not be doing it.

No, sorry. We don’t know enough to say this. What were the options when configuring ext4? What record size is grafana using? Mariadb or postgres? Is there contention with another db? Is there a mounted NFS path in the same ext4 partition with weird block settings? There are lots of legitimate config reasons for op’s problem, but there are so many parts of op’s setup we don’t know, we similarly can’t possibly know what the issue is.

Swappiness works as intended in most default scheduler configurations and seeing high swap is a symptom, not an issue. It is almost always more productive to find out why the swapping is happening.