Essentially, yeah.

Elves’ spirits either linger in Middle Earth or go to the Halls of Mandos in Aman. After some period of time, they can be re-embodied if they choose.

The souls of men did not linger, they were called to the Halls of Mandos upon death. Their souls would stay for a while in Mandos, separate from the elves, until they departed the Halls to only Eru knows where.