Banana physicist here.

This is actually due to a really interesting phenomenon called banana dilation. Although they may appear different sizes to the eye, bananas distort local spacetime such that they are all physically identical dimensions. This makes them a perfect and consistent measure of size.