You assume that it will be ever the self driving software in charge or the button pusher taking the wheel. You did not consider that the button pusher might have a foot on the brake, but instead of taking the wheel he might have to enter some commands.

Like the case where there is a road block ahead and the button pusher has to evaluate whether it is safe to move forward or not, but he wouldn’t take the wheel he would tell to the driving software where to go. In similar cases he would have to decide whether it is safe to pass aside an obstacle or stop there. Even in case of a burglar trying to get on board he would have to call the police and then give some commands to the driving software.