Comment on Uses for local AI?
Amongussussyballs100@sh.itjust.works 1 month agoHow well does the M1 chip keep up? What size models are you running with it? Interested in getting an M1 laptop and I am curious.
Comment on Uses for local AI?
Amongussussyballs100@sh.itjust.works 1 month agoHow well does the M1 chip keep up? What size models are you running with it? Interested in getting an M1 laptop and I am curious.
thirdBreakfast@lemmy.world 1 month ago
I mostly use starcoder2 with Continue for code autocomplete, the big deepseek coder is a bit slow (I can feel it thinking), but it and the regular llama3 are good for chatbot type programming questions.
I don’t really have anything to compare the M1 performance to. I guess the 8GB models output text a little slower than the web versions of the same models, and the 4GB ones about the same. Using ollama in the terminal, there’s sometimes a 0.5-2 second pause before it starts outputting. Not with phi3 though - it’s surprisingly snappy for the quality of answers.