Start by using a 3rd party contacts app (or dialer+contacts as they usually come, at least on Android). Google’s Contacts app only works with Google.

The 3rd party app should let you explicitly select which sources of contacts you want to use. After you set up DAVx5 you should see it available as a source.

The app I use (True Phone, com.hb.dialer.free) shows a list of all sources under “Settings > Contacts > Contacts to show” and you can check/uncheck the ones you want.