I would only want the phone to listen when I actually ask it a question, not 24/7.

If the phone does not listen 24/7, then how does it know when you are asking a question? It should discard all information until the wake up word is called in theory. Only way it could work if you have to press a button to start listening to your question. This was the case in the past, however people wanted to ask questions while showering or something since they introduced this “improvement”.