Generally speaking it will be fine. SSH will also refuse keys with open permissions so you would notice if it was wide-open to other users of the device.

But you know if you are running random code or AI harnesses as that user it can be at risk. Or if you copy around the key all over the place it is more likely to leak. But generally speaking you are secure by default, just don’t do something dumb with the key and you’ll have no problems.