You don’t need that assumption. Your assumption can just be “the person and vessel don’t diverge significantly over time”.

Then, if you treat velocity as a vector and compute the average velocity vector over time, you’ll have a pretty close estimation to the vessel’s velocity vector.

After all, if those two average vectors (vessel’s and person’s) were to differ much, they would end up in different locations.