In principle, yes, and I believe a few small hobby projects have attempted to do this. However, interest developing a custom Smart TV platform tends to get siphoned away into a project where the output is displayed on the TV simply because it’s easier to develop and maintain support across different model TVs.

Why would you develop a custom TV OS that runs on one TV when you could develop it for any mini PC and immediately support all TVs? You’d have to develop your OS to run on each specific TV model which will make it quite hard to reach a critical mass sufficient to attract attention from developers and users alike.

The juice isn’t really worth the squeeze. It’s not like TV vendors are publishing detailed hardware specs and drivers. Writing or even porting an OS is hard. Look at the state of the Android ROM scene, and that’s about as good as it gets when some vendors are actually attempting to open source their drivers. The difficulty is much higher and the interest lower due to the existence of a viable alternative.