Pretty sure that’s an intentional evolutionary trait. Fathers are less likely to question paternity if a baby looks like them, helping to form emotional bongs early.