This is probably not what you’re looking for, but I have two suggestions that don’t really fit your criteria but you might find at least mildly interesting:

1. Playstation Vita. Moddable, retroarch gives you most retro games, and you can stream your PC with Moonlight. It’s a bit small but it roughly fits your price point and is just awesome

2. Surface Duo & Duo 2. A bit out of your price range but with a good expandable controller, I consider this the ultimate Nintendo ds emulator. And I mean, I guess android is sort of like Linux? lol

The original Surface Duo is dirt cheap on ebay and I enjoy it

I am certainly interested in something that fits your criteria more closely. But it most likely doesn’t exist out of the box. You’d probably have to hobble it together yourself with a Raspberry Pi and other hardware