If you have one on one meetings it might be a good opportunity to ask (depending on the circumstances the other manager told you the feedback).

If you want to leave, then it’s enough reason to leave, but I would strongly recommend finding a new job before quitting.