Of course it’s silly.

It’s also silly that people get upset at a bunch of Asian countries who still use the swastika. But a lot of people outside the cultures instantly think of Nazis. In the same way that remembrance poppies aren’t really a thing outside the commonwealth. So to some it’s a symbol of racism.