This blog and the Wikipedia are good starting points.

It’s useful to note that there were multiple attempts to go the “Oops! All kana” route or use romaji, but for a variety of reasons cultural, political, and linguistic, those didn’t pan out. Writing systems are deeply informed by a specific historical and social context, and what at first seems like irregularity or unneeded complexity, are actually the traces of that history marked on the language.

As for issues like why katakana is used for non-foreign words too, I thinks it’s best to think of language feature less as strict rule followers and more like a species in its ecological niche. Katakana is very good at rendering foreign words in Japanese, but if it finds some unfilled gap that isn’t being better filled by some other feature people will use it to to fill that gap too. When the semicolon was developed in English no one imagined at the time we’d use it to do this ;-) but here we are.

Take all this with a grain of salt as my expertise is in language more generally and not Japanese specifically.