I’ve run small models (a few Gb in size) on my steam deck. It gives reasonably fast responses (faster than a person would type).

I know that they’re far from state-of-the art, but they do work and I know that the Steam Deck is not going to be using much power.