Perhaps they assumed that you felt obligated to speak to them, and they felt the need to absolve you of that obligation. That would be the literal meaning of the phrase.

A different interpretation could be that they didn’t want to continue the conversation. In this scenario, they would prefer to be silent, and by suggesting that, “You don’t have to do this,” they have given you a sort of symbolic agency in ending the conversation, implying that both of you would rather be silent.