Someone with the flick of a piece of wood can just murder you outright, with you having no way of defending against it. Before they learn to control it, the magic of the children just “flows” in uncontrollable ways. Maybe doing something silly like coloring hair, but maybe also destroying things and hurting people. For a non magical person in an entirely unpredictable way, without any possibility to intervene or repair the damage as easily.

Keep in mind that there is no mention of the Dursley’s ever receiving any sort of guidance on how to deal with a magical child. They are just expected to figure it out by themselves. Imagine you would have to deal with a special needs child, just that nobody is giving you any information, you can not reach out to anyone about it, you must keep it a secret as otherwise you get booked in the closed psychiatry and on top of it all the child at any point may burn down your house.