Right, this is the correct answer. Even with a very tall ladder, the force is relatively small. If the ladder is well anchored, you can demonstrate this by standing at the top, holding on tightly, and pushing on the wall to see how hard you have to push to move the top of the ladder away from the wall a little.

Also worth mentioning that they sell covers/pads for the top of the ladders to prevent denting the drywall, but you can also make them from a piece of foam pool noodle.