The cache usage being higher on a server than a laptop seems fairly likely as more disk accesses are occurring. When data is loaded from disk, the linux kernel will keep it in memory even after the original requester is done with it, on the principal that it might need it again. This is the cache you’re seeing. If the kernel detects high memory utilisation it will free cache pages before anything else, so you actually want to see near 100% memory utilisation all the time, as it means fewer accesses to disk.

The swap usage is harder to diagnose without more detailed diagnostic work, but you can see from the graph you posted that, even what swap is being used, you have a little free memory. This may well be the kernel preemptively moving little used memory pages to the swap cache so tgat they can be evicted from memory quickly if needed. You could investigate thus by adjusting the ‘swapiness’ value closer to 0, to see if that delays the swap usage and reduces the peak.