While pure silica gel should not be very toxic, colored silica gel has compounds that produce that color change.

You’ll notice that silica gel is not sold as “pure”.

The thing is that none are food grade, so it’s perfectly legal to have impurities, and from what I have read, the coloring agents are toxic. Usually not worrying in the amounts used in the hobby, but still, what I warn people is about heating, breathing, etc products or materials which produce suspensions, aerosols, vapors for which we have little research or data.