Comment on Hackers can read private AI-assistant chats even though they’re encrypted
PlexSheep@feddit.de 8 months agoThat actually is really really interesting. Thanks for giving the tldr. Do token lengths vary that much?
Comment on Hackers can read private AI-assistant chats even though they’re encrypted
PlexSheep@feddit.de 8 months agoThat actually is really really interesting. Thanks for giving the tldr. Do token lengths vary that much?
kevincox@lemmy.ml 8 months ago
Absolutely. They are sort of a compression scheme so the tokens contain different numbers of characters based on how frequent that string is. So common words like “the” will typically be one token, or maybe even common phrases like “I am”. On the other hand rare punctuation such as “~” may be its own token. There will also be tokens for many common prefixes and suffixes such as “non” and “n’t”. The tokens of each model are different but they definitely vary in length.