You either use disk encryption (and a lot of other things as well) and accept these concerns as part of the deal or you dont use it. That’s why actual security involves threat modeling and mitigation. Encryption by itself gives little more then headache.