I read some random theory on the Internet years ago that said we may well have memories from earlier, but because of how much the brain changes during development they could basically be encoded in a different file format than what the brain at a later stage of development knows how to decode. No idea where I came across that, or if it was supported by any science or was just a shower thought, but interesting to think about.