they are very different in my mind. perhaps because i first came across them in their respective contexts through reading.

even when speaking, to me, lose rhymes with booze and loose rhymes with goose.

this has never been a problem for me, personally.