I once did some fun boxing with a friend of mine. He punched me just in my arm without any effort and it was enough to make me stumble. He is around 10-15cm taller than me, and around 20-30kgs heavier. My arm was blue for weeks.

Weight absolutely matters. It’s simple physics (kinetic energy)

Also you can compensate a lot with training. I‘m also in this picture. But since I do regular cardio, I can run for hours and move my dad body the whole day without getting tired. With a bit of strength training I am able to build some serious invisible strength. I wouldn’t underestimate a guy with a bit of body fat.