I think you might be confused about the difference between giving the LLM access to your stored conversations during your session and using OpenAI using AI to search your stored conversations.

What the LLM has access to during your session changes nothing but your session.

It’s not some “I, Robot” central AI that either has access or doesn’t as a whole.