The hashes aren’t tied to a specific identity, they’re just a simple way to convert a set of data into a unique, fixed-length string. The hashes could be rewritten, but without access to the site’s private key to sign them, they wont validate against the public key.

The key itself is also signed, so even if an adversary tricked you into downloading a fake public key, the fake key wouldn’t be signed by my personal key.

The important point is that it’s a layered system, rather than a single point of trust.