brucethemoose

Duct it!

I have a 400W 3090 with zero case fans.

Its always cool, because I bought like $10 of weather stripping to “seal” its intakes against the edge of an SFF case. So it’s always sucking in ambient air:

Image

Image

You can also severely undervolt a 5090 to like 250-300W, with very little performance impact. Honestly its stock speeds are kind of crazy.

First of all, I mean zero offense with any purchase decision. A 5090 is very good.

…But if I were paying that kind of money, I’d probably get a 4090 and a new motherboard/CPU instead. Maybe a used DRR4 threadripper system.

Hybrid (CPU + GPU) inference is where it’s at these days. It opens up a whole world of huge MoE models. And having a big CPU is much more important for that than having a 5090 instead of a 4090 or something.

Be aware that you pay a significant performance penalty for going over occulink.

Even my old 3090 got a huge jump going from a PCIe 3.0 x16 riser to a PCIe 4.0 one.

• It’s just under 300B, trained at FP4; absolutely the perfect size for servers with 128GB-192GB CPU RAM to spare.

• Its fast. I’m getting 17 tokens/sec on a single RTX 3090 GPU, all experts offloaded to RAM; for a 300B model this smart, that’s crazy fast.

• Its attention mechanism is cutting edge, good for long context without too much processing time.

• The benchmarks for coding/agenic usage are absolutely bonkers, within margin of error of frontier models or Deepseek Pro in some cases: huggingface.co/…/DeepSeek-V4-Flash-0731

• …Though I don’t put much stock in benchmarks. And I haven’t tested it enough to tell you if it lives up to that hype for specific use cases.

• Deepseek also publishes its base model. That means I can “unfry” the model with a merge if I have to.

I am afraid of the the model being “overfit” to coding and agenic stuff.

For reference, my previous favorite model was Xiaomi MiMo V2.5 310B. It benched well, but it also smelt “smart” outside of benchmarks, like in knowledge of trivia without tool usage/internet access or comprehension of weird questions.

I know this isn’t everyone’s cup of tea, but I’m excited about Deepseek V4 flash. It’s (for me) the perfect size and architecture to self-host and LLM.

My box is chugging through a queue:

• Figure out why my swap is going crazy, and how to ban processes from it [Done].

• Make an ik_llama.cpp iMatrix for Deepseek V4 [Done].

• Figure out what quantization isn’t working [Done].

• Make a test IQ2_KL/MXFP4_R8 quant to see how it does squeezed onto my box [in progress].

• Test. Inevitably troubleshoot the dozen other things that go wrong. Figure out how much spare RAM that leaves me.

• Make a higher quality IQ3_KT quantization. This will take all night.

• KLD test both of them vs the full precision, to quantify quantization loss. Likely an overnight task, too.

• Try merging the new model release with the base model, 50/50, for a less “deep fried” model. imatrix, quant, test.

It’s not linux’s job to be an ethical gatekeeper, within reason.

Let me ask you this; if you’re from a certain country, from a certain background or whatever, should you not be allowed to contribute the kernel? What about its usage; should certain entities not use it?

Devs of open source projects find out others doing things they don’t like with their code. Or find major contributors have backgrounds they don’t like.

Happens all the time; drama ensues. But at the end of the day your permissively licensed project is out there, the work on it is done by whoever’s already contributed.

I’d assert LLM agent usage is the same. Maintainers don’t have to like them, but it’s not really Linux’s job to judge who (or what) wrote the contribution; just if the code is good, or not.

Now, should LLMs be maintainers? Hell no.

Should they blindly contribute code? Hell. No. Hence the policy requires a human to be responsible for it: docs.kernel.org/process/coding-assistants.html

Among all the horrid types of racism in the racism zoo, this one makes the least sense to me.

…Have they ever met a “race-mixed” person? They’re freaking gorgeous. Like, how can ostensibly justify this?

All true.

I’m just saying I can understand how people got in that mindset. I think the hate should redirected to the actual perpetrators: the Tech Bros.

To be fair, many have had their workplace enshittified by AI, or managers with AI psychosis.

Or fired over it.

…I think it’s understandable for people to be pissed, and take an absolutionist position, as the leads who see it as a tool seem to be few and far between.

+1

Proof is in the puddin’.

If people think this will poison the kernel, well, see if it happens. If it does, fork it from an earlier point.

It’s not just Linus though. This is the exact position of many devs all over the world, ones who are still doing a good job and have clearly not fallen into AI psychosis.

+1

Especially in regards to the nonsense that transformers LLMs will somehow lead to AGI. That is complete nonsense. The bubble is going to pop, and it’ll become a cheap utility.

Meanwhile, algorithmic optimization is basically the root of many world problems.

Also, if you have a more specific question about memory types, I can try to answer them.

But honestly my knowledge getting dated too. For example, I don’t know much about the specifics of GDDR7 (RTX 5000 memory) off the top of my head.

Like a primer? I don’t know anything off the top of my head, but all this stuff is literaly 1st-year computer engineering in college. The basic of ISAs, circuits, low level programming and computer design are all kinda background you need to understand why RAM trace length is so important.

There are definitely college courses you can look up and read for free.

Most “guides” you’d find on Google repeat outdated information; watch out for that. That’s why the Apple “memory on the CPU” thing keeps persisting; most discussion isn’t even updated to account for CAMM modules. But ECE focused magazines are good about covering and explaining commercial memory tech:

semiengineering.com

www.eetimes.com

Yes.

Actually, it’s everything: en.wikipedia.org/wiki/Random-access_memory#Memory…

…But also, your friend is wrong. And its complicated.

First some background. There are a couple of “regular” types of RAM basically every product uses under the hood. And every one is a set of tradeoffs: bandwidth, latency, capacity, power efficiency, trace lengths, expense, and so on. A few of types:

• “Regular” DDR. This is what you see on sticks for desktops, laptops, and servers.

• LPDDR. Its power-optimized at the expense of some latency and trace length, but its specs are otherwise similar to DDR. This is what smartphones (and Macs) use.

• GDDR. Optimized for bandwidth and cost over everything. Guzzles power. Used on gaming GPUs.

• HBM. Low speed, requires extremely short trace lengths, but low power and latency. Uses an exotic bus with exotic manufacturing techniques. This is what you see on more exotic server accelerators.

But the packaging is also really, really important too. For example, regular DDR5 can come in:

• Unregistered RAM sticks, for your desktop.

• As registered RAM, for servers.

• As SODIMM, for laptops.

And these all have long traces, since they have to come in sticks. They’re “far” from the CPU. Laptop SODIMMS, in particular, have a really poor path to the CPU, and start to run into huge scaling issues at DDR5 speed.

…In other words, laptop DDR5 SODIMMS are particularly slow, because the socket they sit in simply cannot keep up with their speeds.

As for LPDDR5: its specification assumes a shorter, higher quality trace or “wire” to the CPU, hence it can only come soldered, and sitting right next to the CPU:

Samsung graphic

But electrically, it’s almost the same as DDR5! It’s not wider-and-slower like HBM, or more tweaked like GDDR5: it’s just regular old cheap DDR5, massaged to work better soldered to the motherboard.

This is what Macs use.

They’re not using “special Apple RAM” like is the popular perception. They’re using smartphone RAM. Its a bit faster becase it is “closer” to the CPU, but this is misleading, as its nothing like more exotic standards that really take advantage of that proximity. If you read this, its much closer to GDDR5 than HBM: graphicscardhub.com/gddr5-vs-gddr5x-vs-hbm-vs-hbm…

In practice, Apple RAM is faster and “closer” to the CPU, but its more because of business design choices than a particularly exotic memory specification:

• Apple uses very wide buses for their CPUs. Typical laptops are 64-bit or 128-bit, Apple stuff can be 192-bit to 512-bit. This is more expensive, and you pay for it in the sticker price.

• Regular laptops use swappable SODIMM sticks, which have a tremendous speed cost. Apple doesn’t have to worry about this.

• Apple pays a premium for fast, binned LPDDR5, regular OEMs do not.

But it doesn’t have to be this way.

Some laptops already use swappable LPCAMM modules, which for all practical purposes are just as fast as Apple’s RAM, and just as “close to the CPU” from an electrical perspective:

LPCAMM module

Framework was very close to implementing this for AMD’s 256-bit Strix Halo CPUs. They could quite work out the electrical gremlins before release.

TL;DR

Apple RAM is “closer to the CPU” in the existing market, but there’s no technical reason PCs have to be any different.

Apple uses off-the-shelf smartphone RAM, basically.

An AMD Strix Halo laptop, with 2 LPCAMM modules, would be just as fast as Apple memory, and just as “close” to the CPU. Some PCs already do this on a smaller scale.

So strictly speaking, your friend is correct. But its misleading. All that really matters is the memory specification laptop/desktop manufacturers choose.

I am taking another stab at Rimworld, after a long hiatus. Putting together a mod pack for multiplayer.

At risk of going off-topic, Halo’s fall is really sad.

Like, I still imagine everything Infinity could have been as a franchise-reboot, clean slate, high budget, open world game, and it just felt so… underwhelming? Conservatively developed, yet not quite right either?

It feels like it could have been a Halo 3 fan mod, like they didn’t even leverage the format. I know that’s technically not true, but still.

That makes sense. The host is pretty unreliable.

Which is the second issue. No image hosts even support JXL or AVIF, except catbox.moe. It’s infuriating.

What platform is that, out of curiosity? Im pretty sure everything should load an AVIF.

Om Android, or iOS?

JXL is working alright for me.

As an example with a lot of dynamic range, here’s a JXL:

JXL

AVIF:

AVIF

Both render in my desktop and iPhone browsers, just fine. I bet at least one renders for you. And I made them from RAWs from a really old camera!

The problem is, as you say… arbitrary lack of support. As an example, I can’t upload either file to Lemmy. Brand new social media software, and it doesnt’ recognize JXL or AVIF as valid image types, even though they should render just fine? Most image hosts wont take JXL either, hence I had to upload them to litterbox since catbox is down!

An HEIF, on the other hand, has basically 0 support outside of Apple:

HIF

All the HEIC files from my camera are still busted :(.

To be fair, its a tricky issue. Its camera makers’ fault for using a format no one else wants to touch, and rendering them as HDR files instead of SDR with gain maps, as is standard practice for smartphones.

…But still, its annoying. They render fine on my iPhone, on Windows, or KDE Linux, out of the box. But they’re completely garbled in Immich :(

Vibecoded self promo is a growing, specific spam problem though.

And a appreciable fraction of Lemmy/Piefed is “anti AI absolutist.”

I think that’s pretty unique.

Be sure to try the ik_llama.cpp fork. Basically, it specializes in MoE CPU offloading on Nvidia cards, and more efficient quantization types than mainline llama.cpp:

github.com/ikawrakow/ik_llama.cpp/

And see this repo for specific 3090 configs: github.com/noonghunna/club-3090

Honestly I should just write up my general setup in this community too.

So is this a leak?

Because, with a cursory glance, it doesn’t always look like spam.

A classic example I see starts with “I built a…” in the title, has a wall of text in the description, and actually promises to do something interesting. Only upon deeply inspecting the code (or trying it yourself)… it becomes clear it’s hallucinated nonsense.

And it’s not always malicious, either. A lot of devs get deep in AI psychosis as truly believe there building something revolutionary with their vibe coding agent.

And sometimes these projects are interesting!

Hence it would be EXTREMELY helpful to have this tagged, up front. To me, an [AIP] is gigantic red flag to warrant extra caution, but not necessarily a smoking gun, and would help “regular” homebuilt projects stand out from the vibecoded ones.

And [AIT] is just nice to have. Some users don’t want to see any AI in /c/selfhosted, period. It gets reported as spam because people interpret it as that, and this gives would prevent that while giving those users a way to easily filter them out.

Normally I’d agree, but the tagging rule won’t affect the majority of posts. I think it’s an acceptable complication, in this case.

Especially with how much vibecoded spam is in the horizon.

Vibecoded spam is deliberately engineered to look “high effort,” so even with the vagueness of such a rule, it wouldn’t cover the spam so well.

Failure to provide a disclosure after using the tag would mean removing the post. It could be locked, but I would have to assume the majority of the spam-type postings that happened to make it past the rule 7 criteria are the ones who will not provide the requested disclosure. I think it makes for a good filter this way, but please comment if you think otherwise.

Sounds reasonable to me!

I think the major choice is for y’all (the mod team), as enforcing a tagging system is going to increase the moderation workload.

I have no recommendations for a bot.

…You could use an embeddings model for a little extra automation though.

This is a pre-LLM thing, but basically you could feed a script new untagged posts, use a embeddings model to compare the text of their bodies to a keyword (“AI”?), and spit out a number as a rough “similarity” metric. If it’s above a certain threshold (eg if the post seems AI related), send a message to the moderation team to check it, or maybe even post a rules reminder in the comments.

And FYI, embeddings models are tiny, so it doesn’t need extra resources to run or anything.

I have a single 3090!

And I have 128GB RAM. So the best model I can run is MiMo 2.5 (a 300B model) at around 10 tokens/sec, using hybrid CPU inference.

…But that’s the worst-case scenario, for speed. It’s an IQ3_KT quant (with is a high quality quantization type but very slow on CPU), with a model that barely fits in my RAM+VRAM combined, with no DFlash or any kind of speculative decoding turned on. I could tune it to be much faster, but I mostly just want “max quality, fast enough.”

For speed, or prompts with lots of thinking or context, I just run Qwen 3.6 27B now. That would fit in your 3090 no matter how much CPU RAM you have, but you just have to be smart about the backend and quantization you pick. If you just use Ollama, it’s gonna tell you it won’t fit, or use some horrible default that spits out garbage.