I think it depends on whether the sound difference actually makes a phonemic difference in the source language—like, would a native speaker hear it as an oddly-pronounced version of the correct sound, or as a distinct, different sound in that language?