github.com/vndee/local-talking-llm

Seems like it would be a good place to start. You’d need to write the bit in order to send the output to a voip service and receive the input from the same service.

If you could get that going in a container you could spawn a bunch of them on VPSs (finding ones that have the hardware to run local AI would probably be expensive, probably better to use a hosting service if you’re going to scale this).

I’m sure there are more conversation agent frameworks that people have built (it’s a pretty simple loop to create), but if you wanted to get started this isn’t bad.