The interference pattern disappears if anything becomes entangled with the which-way information at all. You can replace the entire measurement device with a single particle that interacts with the particles at the slits in such a way that it becomes perfectly correlated with the which-way information that the observer has no awareness of (such as if a moat of dust interacts with the particle because the experimenter did not isolate it well) and that is sufficient for the interference pattern to disappear.