vhstape

@vhstape@lemmy.ml

This is a remote user, information on this page may be incomplete. View original ↗

I make computers

⁨Comment⁩ on ⁨[AIT] what is your selfhosted local LLM stack?⁩ ⁨⁨20⁩ ⁨hours⁩ ago⁩:
I’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG.