Usually I stop after 2 or 3 attempts.

You could also ask him directly. Maybe he’s generally not interested in lunching together, or he was just too busy to respond, or whatever. These questions can be awkward, but also deepen a friendship. A risk-reward mechanic in the game of life.