Comment on How can i teach myself to contribute to these projects?
DupaCycki@lemmy.world 6 hours ago True, it’s always important to be careful with chatbots, even when used for simple tasks.
From my experience, they work fairly well when asked to explain code or error logs. For reference, I mainly tested Gemini and Deepseek. Apparently Claude is the best for coding, but I haven’t really tried it yet.
Personally I’d avoid both Mistral and Copilot. I used to work on Copilot, so unfortunately had a lot of exposure to it. Tried Mistral for a while too.
These two are the worst LLMs I’ve used so far. Honestly probably worse than local Qwen with 27-35B parameters. I feel a little bad hating on a European company, but unfortunately it’s way behind competition. At least from my testing.
Interesting. I tested ChatGPT and Mistral and oh my god, ChatGPT was abysmal. Hallucinated way harder. Mistral is okay if being used on highest settings.
Did you create an account with Mistral? Without it they only expose a low-end model to the web, which isn’t properly declared last time I checked. Can be the free plan, I guess they just want to avoid their infrastructure being hammered.
Personally all those Silicon Valley pricks can offer whatever, I don’t even have a Google account anymore and couldn’t be happier with that. Not saying Mistral is optimal, but at least they’re not part of the current US swamp. And I at least somewhat believe they actually read the damn GDPR. I’m totally fine with it not being the “cutting-edge”, especially given the cutting-edge cuts straight through culture and people’s lifes on purpose and with maximum brutality, and I seriously liked to see their image generator being way worse as well as Mistral refusing to open sites that banned it in the past (if they changed that behaviour please let me know). Not to mention their site isn’t even remotely as infected with trackers and other junk as the others, at least that’s what I’ve read (forgot to bookmark that hackers’ analysis I’ve read a year ago unfortunately).
Yes, I did make a Mistral account. Forgot to mention that I tested all of the mentioned chatbots on free plans, except Copilot, which I had the highest tier plan for, as part of my job. Needless to say, even on the highest tier Copilot performed the worst. Congratulations, Microsoft.
I do appreciate Mistral being European and maintaining better values than American competition. That’s why I feel bad rating it so low. Although as you said, many tasks that LLMs are useful for don’t require the cutting edge at all.
In fact, Mistral Vibe (I preferred the name Le Chat, but oh well) is probably more than enough for most people. It’s just that availability of much better models makes it look worse than it is.
There’s also the question of system prompts. ChatGPT has been known to work considerably better with a good prompt setup, to the point where it’s not one of the worst chatbots anymore. I was only testing the defaults. It’s entirely possible that setting the right system prompt for Mistral improves it noticeably.
At the end of the day, however, if I can run a small, open-weight model locally with comparable results - using a huge, cloud-based and closed model makes no sense to me. I’m not connecting to any cloud unless I really need to.
Alibaba is releasing Qwen 3.8 27B in a little over 3 hours from now, so I’ll have to try that too. Though I’ll probably wait for fine-tuned community releases to make a more detailed comparison.