Move away from port 22, and 90% vanishes. Move it up to a port in the five digit range, and you will rarely see them.