Assuming that there is at least some amount of slippage between the wheel and ground, it seems to me that you’ll need to regularly check the ToF sensors anyway. I’ve found that encoders are fantastic for a lot of things, but not so much for measuring distance because of the problems you’ve described. Perhaps a recurring local check on a reduced set of points to verify location then forward the full cloud less often for further remote processing? It really sounds like you have a tradeoff depending on whether you value accuracy of location or accuracy of wheel rpm (analogous to speed). Using both would give you a nice way to calculate the ideal motor rpm to minimize slippage in a surface agnostic way.