Comment on Descentralized AI book reading server
Laristal@lemmy.dbzer0.com 1 week ago
You’re thinking of retrieval augmented generation (RAG) with a vector database. Its something that’s being actively developed. I haven’t had time to dig much into it, especially currently. But the terms themselves should give you a starting point.
rumba@lemmy.zip 1 week ago
Rag is fucking awesome, But in its current state it can’t handle unlimited amounts of data. On consumer machines I think you can throw around 100 megs at it before it starts losing it That’s quite a lot of text, but not really a decent collection of books. They might be able to get away with separating the books into categories and adding them as different knowledge bases. They’d have to select which knowledge base they wanted to ask but if they could keep the size down it might work relatively well
I fed mine about a year’s worth of slack traffic from work. I would ask it how many times people had trouble with a certain system. It would say three, meanwhile there were 500 tickets in the system of people having trouble with it.
No if I asked it about those three things it would have great detail. I can even ask it for sentiment of people that were talking about it It would recognize reasonably well if they were upset, understanding or angry.
Cyberflunk@lemmy.world 1 week ago
What are you talking about? RAG is a method you use. It only has limitations you design. Your datastore can be whatever you want it to be. The llm performs a tool use YOU define. RAG isn’t one thing. You can build a rag system out of flat files or a huge vector datastore. You determine how much data is returned to the context window. Python and chromadb easily scales to gigabytes, on consumer hardware, completely suitable for local rag.
rumba@lemmy.zip 1 week ago
I explained what I did, and how it worked.
generally, this: www.youtube.com/watch?v=qV1Ab0qWyT8
the numbers came from my experience, ymmv.
SchizoDenji@lemm.ee 1 week ago
I think there are graph based solutions for rag, which ideally should have the books at leaf nodes, and their overlapping summaries as parent nodes.