I'd say on the Hardware side, degoogling is a pain. Android devices are not very 'open'.
I think graphene is the only alternative that's easy to install and that only works on a couple of devices. For others like linage there are a few more devices but some of the install procedures are shonky and can leave you with unlocked bootloader and stuff like that. Scares off many people I'd think.
I think a device that was designed to allow you to install an OS and is not basically locked down is attractive.

Put another way, you can fork android all you like, but what are you going to run it on, and how do you install it?
If there was a device that made it easy , it might get popular enough to attract more interesting open source innovations.

On the software side , once you ditch google there's also way fewer applications. F-droid is ok; but being able to use one of the major distro with an ARM repository would give a decent amount of stuff - albeit not optimised for touchscreen.

I think there's an options also some technical stuff about how much call and cellular data has to go through your cellular network, and whether bypass or switch off is an option - I think that's for the real privacy people - I don't really know about it.