wouldn’t diffuse light be what it’s going to be best at? While it’d be worse on a sunny day when there is an optimal single direction for the light to come in?

No. Concentrated solar requires perfect alignment, dual axis tracking, to the sun. diffuse light does not concentrate.

A reasonable alternative design would be cheap ordinary PV cells with outward bubbles instead of inverted parabolas that would capture off axis light better on a fixed tilt.