Sometimes people will say “That person’s name!” or “Those group of people!” in anger. “That Donald Trump! How dare he claim immigrants are eating pets?” to give you a current example.

When spoken of a family member or mutual acquaintance with a chuckle, it means more like “That person has some strange quirk but what can you do? We still love him.”

For example, you might hear “That dog! Always chasing his own tail.” So I think this is likely what you were getting from that conversation? It’s certainly not a criticism of your use of the word “people”.