they are very different in my mind. perhaps because i first came across them in their respective contexts through reading.
even when speaking, to me, lose rhymes with booze and loose rhymes with goose.
this has never been a problem for me, personally.
oh, no, no, no! booze and a goose should never go together!