We use the word differently. In the past I think we used it more as you do, because “going to town” had the connotations of going to a big city.

“Town” in American usage can mean anything from a small urban center (like under 10k people) to an incorporated municipality that has only a post office and tons of farms around it.

Basically we don’t say “village” here. So town is the smallest word we have. But it has a big range.