Humans aren’t that homogeneous when it comes to immunity though.

When Europeans colonised Americas they brought germs that decimated Indians. There’s plenty of evidence that similar thing happened in Europe too when Yamnaya / Corded Ware culture expanded because they replaced existing people very quickly and without much violence which also coincides with burial sites teeming with black plague but I digress.

You also have to account for unused immunity being possibly eroded with time because there’s no evolutionary pressure. This is why we need to be mindful about vitamins for example - we got used to them being available in the environment and our bodies can no longer synthesise them. Dormant functionality isn’t guaranteed to carry over through millennia.

40k years ago is also when last Sapiens and Neanderthal admixture happened which was a bit of big thing for our immune systems to give a sense of scale here.