You should always use a guide and through every distinguishing feature. AFAIK their spores aren’t toxic, just allergenic. But you wouldn’t be getting then when they’re sporulating.

But you’re right, the fact that young amanitas are a lookalike is a good reason to avoid them. Giant puffballs don’t usually have that problem due to their sheer size.