Submitted 14 hours ago by SuspciousCarrot78@lemmy.world to selfhosted@lemmy.world
…it answers from the attached KBs only. If the fact isn’t there, it tells you - explicitly - instead of winging it.
So you’ve made a FAQ with a LLM interface? I could see that potentially being useful for cooperate “let our bot answer your questions” tools.
But the usefulness of AI isn’t just in “tell me a fact”. Like what would your AI give for "what functions would I use in Python to convert a utf16 string to utf8?
It’s a very nice concept to be able to dump a shitload of knowledge into a folder, look at “processing…” for a few seconds, then ask questions to get exactly what you’re looking for out of it rather than having to go digging through the mound of information and without having to worry that the computer just threw in a few made up facts for giggles. The idea is that the dumping happens over time mostly, allowing you to quickly find buried information from years ago with a few relevant queries.
One thing I’d do with this is dump all my emails into it, from across all my accounts. That might save me having to search keywords in 8+ accounts over 4-5 different platforms every so often…
It also might have been useful in a lawsuit I prosecuted a few years ago. Instead of going through two years of encrypted messages by hand to pull out relevant excerpts with context, I could have exported the lot and just asked for the information. If it worked it could have saved me months (I spent a few hours after work every night screenshotting, dating the screenshots in chronological order, then I’ve that was done I kept a spreadsheet that I filled with relevant excerpts and links to the screenshots, by reading every single screenshot… it was a lot).
I would be super interested to hear if it could do that. I genuinely don’t know, because I haven’t tried it.
If you can export your emails in the correct format, it might actually work. Try a small batch and report back.
PS: you DON’T HAVE TO run >>summ if you don’t want to. You can ask questions against the raw files too.
Wishing you luck!
Correct. Curate your sources :)
I can’t LoRa stupid out of a model…but I can do this. If your model is at all obedient and non-stupid, and reasons from good sources, it will do well with the harness.
Would you like to see the benchmarks for the models I recommend in the “minimum reccs” section? They are very strong…and not chosen at random.
Like the router, I bring receipts :)
Isn’t this just a notes-taking app with extra steps?
“Extra steps” lmao
No. Notes apps store your text. This is a control panel for how a LLM reasons over them.
I commend this but I guarantee you the AI will find a way to lie!
Oh it can try…but you can see it’s brain. That’s the glass box part of this. You can LITERALLY see why it says what it says, when it says it. And, because it provides references, you can go and check them manually if you wish.
Additionally (and this is the neat part): the router actually operates outside of the jurisdiction of your LLM. Like, the LLM can only ask it questions. It can’t affect the routers (deterministic) operation. The router gives no shits about your LLM.
Sometimes, the LLM might like to give you some vibes about things. Eg: IF YOU SHOUT AT IT LIKE THIS, the memory module of the router activates and stores that as a memory (because I figured, if you’re shouting at the llm, it’s probably important enough in the short term. That or your super pissed).
The llm may “vibe” a bit (depending on the temp, seed, top_k etc), but 100/100, ALL CAPS >8 WORDS = store that shit into facts.json
Example:
User: MY DENTIST APPOINTMENT IS 2:30PM ON SATURDAY THE 18TH. LLM: Gosh, I love dentists! They soooo dreamy! <----PS: there’s no fucking way your LLM is saying this, ever, especially with the settings I cooked into the router. But anywayz
[later]
USER: ?? When is my dentist appointment again LLM: The user’s dentist appointment is at 2:30 PM on Saturday, the 18th. The stored notes confirm this time and date, with TTL 4 and one touch count. No additional details (e.g., clinic, procedure) are provided in the notes.
Confidence: high | Source: Stored notes
Yes, I made your LLM autistic. You’re welcome
I’ll look into this, but at first blush this is just mostly tool calling with RAG. This does not prevent a whole host of issues with AI, and doesn’t really prevent lying. The general premise here is to put tight guard rails on how it can interact with data, and in some cases entirely forcing a function / tool path with macros. I am not really sure this would work any better than just a stateful and traditional search algorithm on your own data sources, and would require much less hardware / battery / requirements and would be much more portable.
I like the effort, but this feels a bit like trying to make everything look like a nail.
I’ll cop to that. At a high level it is “tool calling + RAG + guardrails”.
Ok.
But that’s sort of the point: boring plumbing that turns LLMs from improv actors into constrained components.
Addressing your points directly as I understand them -
If you mean “LLMs can still hallucinate in general”, yes. No argument. I curtailed them as much as I could with what I could.
But llama-conductor isn’t trying to solve “AI truth” as a metaphysical problem. It’s trying to solve a practical one:
In Mentats mode, the model is not allowed to answer from its own priors or chat history. It only gets a facts block from the Vault. No facts → refusal (not “best effort guess").
That doesn’t make the LLM truthful. It makes it incapable of inventing unseen facts in that mode unless it violates constraints - and then you can audit it because you can see exactly what it was fed and what it output.
So it’s not “solving lying,” it’s reducing the surface area where lying can happen. And making violations obvious.
I don’t know. Would it? Maybe. If all you want is “search my docs,” then yes: use ripgrep + a UI. That’s lighter and more portable.
The niche here is when you want search + synthesis + policy:
I think an algo or plain search engine can do wonders.
It doesn’t give you a consistent behavioral contract across chat, memory, and retrieval.
Maybe. But the nail I’m hitting is: “I want local LLMs to shut up when they don’t know, and show receipts when they do.”
That’s a perfectly cromulent nail to hit.
If you don’t want an LLM in the loop at all, you’re right - don’t use this.
If you do want one, this is me trying to make it behave like infrastructure instead of “vibes”.
Now let’s see Paul Allen’s code :P
As a handsome local AI enjoyer™ you’ve probably noticed one of the big flaws with LLMs:
LOL I think your AI is lying again. LOL I’m quite certain I don’t belong in the ‘handsome’ category.
if you self host, you belong. That’s non-negotiable
Ah. So -
First prize: picture of you Second prize: two pictures
?
:P
TL;DR: “In God we trust. All others must bring data.”
what the heck does that mean?
This is a quote from Deming, one of the fathers of modern data analysis. It basically means “I don’t trust you. You’re not god. Provide citations or retract your statement”
now i know. thanks
Exactly what it says, what’s confusing about it? The problem with LLMs is that they bullshit; the problem with LLMs bullshitting is that you have to check everything they say, they are not trustworthy - severely limiting the utility. So, don’t trust LLMs. God meanwhile is assumed to be the paragon of honesty, and thus completely trustworthy, so would be the only entity to be trusted implicitly.
This works even if you don’t believe in god, like me: trust nobody, everyone must bring data if they want me to trust what they say.
thanks for that link
“Trust” should only be for the supernatural; Reality can be proven.
Everything else must be proven.
Hi! You are bullshitting us. To understand your own incorrectness, please consider what a chatbot should give as an answer to the following questions which I gave previously, on Lobsters:
The biggest questions in mathematics do not fit nicely into the chatbot paradigm and demonstrate that LLMs lack intelligence (whatever that is). I wrote about Somebody Else’s Paper, but it applies to you too:
This attempt doesn’t quite get over the epistemological issue that something can be true or false, determined and decided, prior to human society learning about it and incorporating it into training data.
Also, on a personal note, I recommend taking a writing course and organizing your thoughts prior to writing long posts for other people. Your writing voice is not really yours, but borrowed from chatbots; I suspect that you’re about halfway down the path that I described previously, on Lobsters. This is reversible but you have to care about yourself.
I have no idea what the fuck you’re on about.
Is this an exorcism? Did you just down a quart of pain thinner? Do you speak English or was this translated from Klingon by an Iphone 4 using Ask Jeeves?
RTFM. Operational terms are clearly defined. Context is defined. Epistemology is not what this is about.
Take your lobster and cook it with some garlic butter.
Additionally, this is “my voice” telling you to “get bent”. You arrogant, patronizing donut.
I don’t know the coding bits at all, but I really like the TL;DR! Is it a reference to the Jean Shepard book?
Not intentionally :)
Sounds interesting. What kind of data can it reliably ingest with “attach”? If I dropped, say, the entire Python docs in there, would it be able to get anything out of that? Or does it need to be minimalistic plain-text statements? How is it actually performing retrieval?
Yes! And, for fun, I forced Qwen to answer your question, after querying it’s own fucking brain (M-E-T-A).
The below is AI generated, using the router, with >>Fun mode (not >>FR…that’s my special gremlin mode). >>f suspends some of the terseness. Let me know if you want to see the non FR version
Yep — with your current code, here’s the literal answer to “what can >>attach ingest, can it handle the whole Python docs, and how does retrieval work?” (based on the files you uploaded).
>>attach actually does (and does not do)>>attach <kb> doesn’t ingest anything. It just registers a KB name into state.attached_kbs so later turns know which KB folders are “active.”
The router’s own docstring is blunt about the intended contract:
SUMM_*.md files.”So the “ingest” step is really:
>>summ new to generate SUMM_*.mdattach)?In the router’s filesystem SUMM pipeline, _SUPPORTED_RAW_EXTS = {“.md”, “.txt”, “.pdf”, “.html”, “.htm”}
pypdf — if missing, the router treats that as a failure/skip with a note (your top-level comment calls this out explicitly).summ.max_input_chars = 120_000).When a SUMM is created, it writes a provenance header including source_rel_path and source_sha256, then moves the original into /original/.
So: you do not need “minimalistic plain-text statements.” You can feed it normal docs (md/txt/html/pdf) and it will produce SUMMs that become queryable.
Yes, it will produce something usable, because Python docs are mostly HTML and you explicitly support .html/.htm with stripping.
But there are two practical gotchas in your implementation:
summ_new_in_kb() walks the KB tree, skips /original/, and summarizes every supported raw doc that doesn’t already have a corresponding SUMM_*.md.SUMM.md template is oriented around “overview / key ideas / steps / tradeoffs / pitfalls” rather than “API reference / signatures / parameters.”
So it’ll work better for conceptual docs than for “tell me the exact signature of pathlib.Path.glob”.If you want Python-docs-as-KB to be strong, the move is: tweak the SUMM prompt to extract API sections + signatures + examples, not just “Key Ideas.”
This is the most important bit: filesystem KB retrieval is not vector embeddings. It’s a lightweight lexical scorer over SUMM markdown blocks.
Concretely:
SUMM_*.md under attached KB roots, explicitly excluding /original/.top_k=8, max_blocks_per_file=3, max_chars=2400).So attached-KB mode is basically: pre-summarize once → then do fast “smart grep” over summaries.
That’s why it’s potato-friendly: you’re not embedding every doc on every query; SUMMs are preprocessed once.
When you >>move to vault, you’re no longer doing filesystem lexical search. You promote SUMMs into Qdrant and retrieve top-k chunks with embeddings + rerank (as configured in README / config).
(Also worth noting: per your ingest module, the intended architecture is explicitly “filesystem SUMMs for day-to-day; Vault/Qdrant only for promoted SUMMs.” )
>>attachdoesn’t ingest raw docs — it just activates a KB folder. The KB mode queries SUMM_*.md files (generated via>>summ new) using a lightweight lexical block scorer (token overlap + heading boosts), not embeddings. You can drop the whole Python HTML docs in a KB and it’ll work, but you’ll want a SUMM prompt tuned for API/reference detail or the summaries will smooth out the stuff you actually care about. If you promote SUMMs to the Vault (>>move to vault), then retrieval switches to Qdrant vector search + reranking.
###########################
Is gut, ja?
panda_abyss@lemmy.ca 14 hours ago
I’ve been building my own LLM tools and yeah, the eagerness to return an answer, any answer, is incredibly annoying. This looks useful.
SuspciousCarrot78@lemmy.world 13 hours ago
Built by an autist to give your LLM autism. No Tylenol needed.
SuspciousCarrot78@lemmy.world 13 hours ago
On the AI slop image. I sez to shitGPT, I sez “Yo, make me ZARDOZ but you know, cute and chibi like”
Enjoy the nightmare fuel
[ZARDOZ HATH SPOKEN]