I also heard this one. My professor in art school said the broom thing may have come from a misunderstanding of a distaff which could look like a broom with the fibres wrapped around the end. Distaffs would have been common and smooth and polished and were likely used for other amusements.