FrankLaskey
@FrankLaskey@lemmy.ml
- Comment on OpenWebUI Release v0.6.0 3 weeks ago:
Looks like it now has Docling Content Extraction Support for RAG. Has anyone used Docling much?
- Comment on Consumer GPUs to run LLMs 3 weeks ago:
Oh and I typically get 16-20 tok/s running a 32b model on Ollama using Open WebUI. Also I have experienced issues with 4-bit quantization for the K/V cache on some models myself so just FYI
- Comment on Consumer GPUs to run LLMs 3 weeks ago:
It really depends on how you quantize the model and the K/V cache as well. This is a useful calculator. smcleod.net/vram-estimator/ I can comfortably fit most 32b models quantized to 4-bit (usually KVM or IQ4XS) on my 3090’s 24 GB of VRAM with a reasonable context size. If you’re going to be needing a much larger context window to input large documents etc then you’d need to go smaller with the model size (14b, 27b etc) or get a multi GPU set up or something with unified memory and a lot of ram (like the Mac Minis others are mentioning).
- Comment on Perplexity open sources R1 1776, a version of the DeepSeek R1 model that CEO Aravind Srinivas says has been “post-trained to remove the China censorship”. 2 months ago:
I think we can all agree that modifications to these models which remove censorship and propaganda on behalf of one particular country or party is valuable for the sake of accuracy and impartiality, but reading some of the example responses for the new model I honestly find myself wondering if they haven’t gone a bit further than that by replacing some of the old non-responses and positive portrayals of China and the CPC with a highly critical perspective typified by western governments which are hostile to China (in particular the US).
- Comment on Linux's Sole Wireless/WiFi Driver Maintainer Is Stepping Down - Phoronix 2 months ago:
I used to daily drive Ubuntu some years ago for work/personal use but have been back on Win 10 primarily for the last 4-5 years. I was considering trying to go back due to how much Windows sucks (despite some proprietary software only being available on it) but remembering the trouble I had with some networking/printer drivers and troubleshooting those issues and then seeing this article Is definitely making me reconsider…
- Comment on 🎵 🎶 🎵 2 months ago:
- Comment on 🎵 🎶 🎵 2 months ago:
Is this what the Leonard Cohen song is about?
- Comment on Are there communities to post videos of police brutality / excessive use of force? 2 months ago:
- Submitted 3 months ago to games@lemmy.world | 17 comments