Definitely not. Test Drive Unlimited 2 leaps to mind, which while it certainly had racing events and racing related content in it, you could also just drive around doing nothing in particular as much as you wanted.

There are several other racing oriented games that nevertheless had open worlds and you’re never actually forced to race anybody in any of them, albeit usually at the expense of sacrificing any game progression and thus having a rather limited vehicle selection. Need For Speed Underground 2 and Forza Horizon, for instance.