I use Nextcloud for that. On Android phone it’s DAVx5. Thunderbird can use the contact via CardDAV, DAVx5 syncs them with the Android addressbook. Fossify Contacts is nicer than the Google contacts app.

The same way it’s done with my appointments. I have also replaced the native Google calendar with the Fossify Calendar here, because it’s less annoying.