This year has had wild model releases and in the cyber world, a massive increase in PyPI supply chain attacks. Combining the two, I thought it was wortwhile chatting about Rank-One Model Editing (ROME) attacks… There’s a range of ways python would get to a model / GPU -> this attack has malicious (naughty) python writing directly to GPU memory. That makes it particularly hard to catch given there’s no spawned processes, or new files or any network calls to hunt for. For those interested - there’s a deep dive blog from cyber org Stealthium that covers GPU-side detection and potential ways to achieve further attribution. Welcome the communities thoughts? Reckon any of these model corruption attacks are in the wild and not yet been caught/seen? (((Also while the example is soviet propaganda, anybody aware of other nations that like both model development, influence campaigns and rewriting history? Just sayin…))) Shout out to the researchers behind the original paper, Locating and Editing Factual Associations in GPT : Kevin Meng, David Bau, Alex Andonian and Yonatan Belinkov. + to the folks at Mithril Security for their PoisonGPT work a while back. submitted by /u/SirJerryLion
Originally posted by u/SirJerryLion on r/ArtificialInteligence