This is a cool idea! This sounds a lot like DANTE and AES67 or AVB are used for in pro audio, maybe they have some ideas on timing sync which I think would be the hardest part as others have said, it is crazy how small of a jitter your brain can hear.