Well, yes, I would say it’s rude if you have no reason to say no. At least hear them out is usually considered nice.

But…

If the person is clearly wanting to sell something to me, or trick me into something, or take advantage of me (typical in touristic places) I would just say “no thanks” and move on.