Actually, a few mm/sec is quite a lot. That could easily be more than enough to deflect an asteroid from hitting earth. That few mm/sec will accumulate over the time it takes the asteroid to travel six million miles.