I can think of a few downsides. Definitely not all equally weighted.

• Poor discoverability. I doubt many people are finding us on Google, but any who do would be unable anymore.
• Inability to send people links. This also hurts discoverability from being able to send people a link for them to check out.
• Problems with linking federation. At the moment, there’s no way to link to a specific post or comment in a way that users of other instances can see it on their instance. So most of the time, the best option is to link it in your own instance, and they can view it read-only in your instance. T his would break that.
• Problems for cross-posts with bodies. Related to the above, when using “cross-post” on a post with a text body, a link is provided to link back to the original. This link will be useless if the instance is private.