I had trouble like this too, so what I’ve done is just give up on using Google photos in any meaningful way.

I still sync to it as a temporary backup, but I periodically copy all media from my device to my local home storage as the true copy.

I have yet to implement a proper open-source alternative for photo organization, but hopefully that’ll be one of the projects completed this year.