There are a few ways this could work, but it hardly seems worth the effort if it’s not phoning home.

They could have an on-device database of red flags and use on-device voice recognition against that database. But then what? Pop up a “scam likely” screen while you’re already mid-call? Maybe include an option to report scams back to Google with a transcript? I guess that could be useful.

Any more more than that would be a privacy nightmare. I don’t want Google’s AI deciding which of my conversations are private and which get sent back to Google. Any non-zero false positive rate would simply be unacceptable.

Maybe this is the first look at a new cat and mouse game: AI to detect AI-generated voices? AI-generated voice scams are already out there in the wild and will only become more common as time goes on.