Microsoft has a library that does exactly what you describe - Semantic Kernel. You can register plugins that do all sorts of things in the real world, and the AI API responds with instructions on how to use these plugins.

I’m very sceptical of LLMs, but this is the closest this technology came to actually being useful.