onlinepersona

It is primarily sponsored by the government of Switzerland 🇨🇭 where it is currently deployed for citizens.

Now this, I like!

Github, not so much

Thank you. That’s good to know.

Thanks for the response. It’s interesting to read about the experience of others.

That’s definitely not my level of disposable wealth/income. I can barely afford one card.

Oh nice. Does that depend on just the model or are there other requirements like CUDA or something?

How do you now run out of RAM? Does it offload to system RAM?

Thank you for that writeup.

Do you know how important the parameter size is? 12b, 24b, 128b, etc. Does it really improve performance or is it like megapixels in a camera: more megapixels don’t necessarily mean a better picture?

And what’s “quantisation”. Context compression or something?

I’ve been considering buying a better card to test models (also want to be personally sovereign), but NVIDIA on linux gives me the jeebies and, last i checked, AMD hasn’t released anything with more than 20GB in a while. In fact, figuring out hardware requirements has been tough and I’m considering just riding this whole thing out. Maybe the bubble will collapse and bring prices down to something reasonable.

What’s the quality of the answers though? And how much context can it hold? I imagine it’s only good for small, short questions, but have no concept of what is needed for that.

I’m assuming you’re using a 12b or 24b qwen model. The ones from deepseek go up to hundreds of billions of params and I can’t tell if bigger number is better or just meaningless posturing.

How many GPUs do you even need to have a usable, self-hosted AI? It looks like he has 6 on his rig. Probably each costs 2k or something. That’s not peanuts. I have a 12GB VRAM card. It probably can’t generate anything in any meaningful amount of time. Which brings me to the question: who is this for?

Regardless, impressive what he vibe-coded there.

I see. Yeah, that was discontinued. The maintainers didn’t have time for it.

3?

Looks like I’ll have to setup BasicSync. I still don’t trust Syncthing-Fork. The way things went down don’t give me any confidence it could happen again but worse e.g the dev introduces something like a “fuck zionists” patch that wipes everything if you’re on an isralean IP. Then I’d be putting myself in danger for using a VPN or TOR exit node in Israel. Not taking that risk.

Thanks for the writeup.