Well yes, if you want a decent search function, duplicate detection, etc., then E2E is just not feasible.

Immich developers confirmed they will never implement it, as it breaks too many features of Immich.