and I’d need to have password access enabled in order to add the keys

Besides the other points, you could just add the public keys directly in the .ssh/authorized_keys(2) file of the server as long as you still have access from another device. That way you don’t have to enable passwords.