I don’t know that they sound that different, but I definitely pronounce them differently in that my tongue is in a different party of my mouth for both of them. When I say clothes, my tongue is near touching my front teeth, where as close is more just below that ridge behind my teeth, so farther back.

I’m from the center of the U.S. for reference.