If you say yes, it can be interpreted as “yes I mind”, if you say no, it can be interpreted as “no, I don’t give you permission”. You always end up having to clarify.