The issue here is that the authorities are letting a piece of half-ass code (Read: AI) decide what is a legitimate threat and, worse still, acting on that determination without question.

Yeah, at the very least, the software should be passing on the statement, and context surrounding it, along with its ‘judgment’, to the authorities, putting all the responsibility for making the call that X genuinely merits action on said authorities.

Of course, that’s just one piece of the puzzle, and not a solution if law enforcement isn’t held accountable when they fuck up.