Nazis co-opted the sign for a Buddhist temple and just flipped it around. A mnemonic would be that the arms on the swastika are pointing right (as in right-wing) and the Buddhists are pointing left.