I tried an in-browser demo of something like this that Microsoft recently took down, and it was an image diffuser running an agent that could contextualize mouse+keyboard or gameplay inputs to behind-the-hood text prompts.

It looked like I was playing a Quake 2 clone, and almost played exactly like it, but weirdly turn-based when I didn’t do anything because it was just an AI generating images. It remembered the corpses of the bad guys I shot and it also kinda remembered the environment it made, including ramps that go up another floor and opened doorways that led to other areas.

Its cool, but not really a good game, very jank and likely resource intensive, which made sense why they took it down.