Comment on If so-called AI is basically just Large Language Models, how come predictive text on my phone is bollock-useless?

0x4E4F@sh.itjust.works ⁨11⁩ ⁨months⁩ ago

• Knusper@feddit.de ⁨11⁩ ⁨months⁩ ago

  I guess, the real question is: Could we be using (simplistic) LLMs on a phone for predictive text?

  There’s some LLMs that can be run offline and which maybe wouldn’t use enormous amounts of battery. But I don’t know how good the quality of those is…

  source

  • bassomitron@lemmy.world ⁨11⁩ ⁨months⁩ ago

    The kind of local/offline LLMs that would work on your phone would not be very good quality. There’s been amazing progress of quantization of LLMs to get them working on weaker GPUs with lower VRAM and CPUs, so maybe it’ll occur, but I’m not an expert.

    I also don’t foresee them linking it up to a cloud-based LLM as that’d be a shit load of queries and extremely expensive.

    source

    • astraeus@programming.dev ⁨11⁩ ⁨months⁩ ago

      OpenAI is probably already handling a significant amount of queries, I think for daily use the LLN should simply initialize a word map based on user history and then update it semi-occasionally, like once a week or two. Most people don’t drastically change their vocabulary in the course of a few weeks

      source

      • EatYouWell@lemmy.world ⁨11⁩ ⁨months⁩ ago

        We’re talking about orders of magnitude more queries if we start offloading predective text like that.

        source
  • SpooksMcDoots@mander.xyz ⁨11⁩ ⁨months⁩ ago

    Openhermes 2.5 Mistral 7b competes with LLMs that require 10x the resources. You could try it out on your phone.

    source
  • Mr_Blott@lemmy.world ⁨11⁩ ⁨months⁩ ago

    That was my next question, thanks!

    Didn’t think of battery use, makes sense

    source
  • Munkisquisher@lemmy.nz ⁨11⁩ ⁨months⁩ ago

    A pre trained model isn’t going to learn how you type the more you use it. Though with Microsoft owning SwiftKey, I imagine they will try it soon

    source

    • SidewaysHighways@lemmy.world ⁨11⁩ ⁨months⁩ ago

      I was so heartbroken when I found out that Microsoft purchased Swiftkey. It was my favorite. Is there any way to still use it without Microsoft involved? Lawdhammercy

      source
  • neptune@dmv.social ⁨11⁩ ⁨months⁩ ago

    I think apple has pitched this for a future iPhone, yes.

    source

    • squaresinger@feddit.de ⁨11⁩ ⁨months⁩ ago

      They’ll probably have to offload that to a server farm in real time. That’s not gonna be easy.

      source
  • 0x4E4F@sh.itjust.works ⁨11⁩ ⁨months⁩ ago

    I guess… why not… but the db is probably huge, like in the hundreds of GB (maybe even TB… who knows), can’t run that offline.

    source
• Arthur@literature.cafe ⁨11⁩ ⁨months⁩ ago

  iOS 17 uses a small gpt-2 based model for predictive text.

  source

  • 0x4E4F@sh.itjust.works ⁨11⁩ ⁨months⁩ ago

    Hm, that’s interesting 👍.

    source
• Kichae@lemmy.ca ⁨11⁩ ⁨months⁩ ago

  The algorithms are the same. The models are different, being trained on a smaller data set.

  source

  • FooBarrington@lemmy.world ⁨11⁩ ⁨months⁩ ago

    No, the algorithms are not the same. Phones don’t use transformer models for text prediction, they use Markov chain-based approaches. Where do you get these ideas?

    source
  • 0x4E4F@sh.itjust.works ⁨11⁩ ⁨months⁩ ago

    Perhaps, I’m not a dev, especially not an iOS or Android one.

    source