I think it needs to work across instances, since we're concerned wit the Fediverse and federation is one of the defining mechanics. Also when I have a look at my subscriptions, they come from a variety of instances. So I don't think a single instance feature would be of any use for me.

Sure. And with the cosine similarity, you'd obviously need to suppress already watched videos. Obviously I watched them and the algorithm knows, but I'd like it to recommend new videos to me.