There are some existing video games that incorporate LLMs or diffusion models. So in one sense, that’s probably very doable.

But I think that it’s probably going to be a slow process. There are probably going to be dead ends. I kind of suspect that early games, even if they’re technically-novel, probably will suffer the same problems that past video games did before they matured. End of the day, a video game needs to be fun, and just throwing a new technology like a powerful graphics card or a fancy natural-language parser or whatever at it doesn’t get you to that fun game. I think that it’s going to take quite some years of game developers iterating to incorporate generative AI stuff well.

That being said, there are some things I’d like to see tried.

• My guess is that it’s probably possible to create to develop some sort of social-media-based video game that generates a choose-your-own-adventure style video game, remembering story branches generated by other users to take advantage of human-assisted creation, and trying to show “top” story forks. Like, make the bar low, use voting or link tracking or something to determine what story branches people like, and show those.

• I’d like to see some kind of system for tracking world state that isn’t purely based on having an LLM look at the entire preceding text for context. That’s a pretty inefficient way to store world state, and implementing game logic at the LLM level is, I think, going to be problematic. Think of something like, oh, a game system like Inform/TADS/glulx-based interactive fiction. You have objects and properties and a game engine that handles tracking them and their interactions. But you try to get an LLM to generate text for those objects.

• There are some games that use diffusion models, either statically or at runtime, to generate illustrations, where the number of permutations would be impractical for a human artist. The ones I’ve seen have been adult-oriented; I don’t know how the field has developed, and there may well be a lot more out there now.

• One thing that I think could be done today is to start using procedurally-generated voices. Generative AI can do pretty decent voice synthesis. Video games are good at doing procedurally-generated text, but if you do that, you don’t get voice audio. That’s not really a game genre, but it’s a way in which one could provide some neat added functionality. I think that to really take advantage of this, there’d need to be a training corpus of text annotated with emotional information and such, but I’ve seen people doing this in a usable form for game mods.