Comment on Frustratingly bad at self hosting. Can someone help me access LLMs on my rig from my phone
BlackSnack@lemmy.zip 20 hours agoBackend/ front end. I see those a lot but I never got an explanation for it. In my case, the backend would be Ollama on my rig, and the front end would be me using it on my phone, whether that’s with and app or web ui. Is that correct?
I will add kobold to my list of AIs to check out in the future. Thanks!
Ollama has an app (or maybe interface is a better term for it) on windows right that I download models too. Then I can use said app to talk to the models. I believe Reins: Chat for Ollama is the app for iPhone that allows me to use my phone to chat with my models that are on the windows rig.
tal@lemmy.today 19 hours ago
For Web-based LLM setups, it’s not common to have two different software packages. One loads the LLM into video memory and executes queries on the hardware. That’s the backend. It doesn’t need to have a user interface at all. Ollama or llama.cpp (though I know that llama.cpp also has a minimal frontend) are examples of this.
Then there’s a frontend component. It runs a small Web server that displays a webpage that a Web browser can access, provides some helpful features, and can talk to various backends (e.g. ollama or llama.cpp or some of the cloud-based LLM services).
Normally the terms are used in the context of Web-based stuff; it’s common for Web services, even outside of LLM stuff, to have a “front end” and a “back end” and to have different people working on those different aspects. If Reins is a native iOS app, I guess it could be called a frontend.
But, okay, it sounds like probably the most-reasonable thing to do, if you like the idea of using Reins, is to run Ollama on the Windows machine, expose ollama’s port to the network, and then install Reins on iOS.
So, yeah, probably need to open a port on Windows Firewall (or Windows Defender…not sure what the correct terminology is these days, long out of date on Windows). It sounds like having said firewall active has been the default on Windows for some years. I’m pretty out-of-date on Windows, but I should be able to stumble through this.
While it’s very likely that you aren’t directly exposing your computer to the Internet — that is, nobody from the outside world can connect to an open port on your desktop — it is possible to configure consumer routers to do that. Might be called “putting a machine in the DMZ”, forwarding a port, or forwarding a range of ports. I don’t want to have you open a port on your home computer and have it inadvertently exposed to the Internet as a whole. I’d like to make sure that there’s no port forwarding to your Windows machine from the Internet.
Okay, first step. You probably have a public IP address. I don’t need or want to know that — that’d give some indication to your location. If you go somewhere like whatismyipaddress.com in a web browser from your computer, then it will show that – don’t post that here.
That IP address is most-likely handed by your ISP to your consumer broadband router.
There will then be a set of “private” IP addresses that your consumer broadband router hands out to all the devices on your WiFi network, like your Windows machine and your phone. These will very probably be
192.168.something.something, though they could also be172.something.something.somethingor10.something.something.something. It’s okay to mention those in comments here — they won’t expose any meaningful information about where you are or your setup. This may be old hat to you, or new, but I’m going to mention it in case you’re not familiar with it; I don’t know what your level of familiarity is.What you’re going to want is your “private” IP address from the Windows machine. On your Windows machine, if you hit Windows Key-R and then enter “cmd” into the resulting dialog, you should get a command-line prompt. If you type “ipconfig” there, it should have a line listing your private IPv4 address. Probably be something like that “192.168.something.something”. You’re going to want to grab that address. It may also be possible to use the name of your Windows machine to reach it from your phone, if you’ve named it — there’s a network protocol, mDNS, that may let you do that — but I don’t know whether it’s active out-of-box on Windows or not, and would rather confirm that the thing is working via IP before adding more twists to this.
Go ahead and fire up ollama, if you need to start it — I don’t know if, on Windows, it’s installed as a Windows service (once installed, always runs) or as a regular application that you need to launch, but it sounds like you’re already familiar with that bit, so I’ll let you handle that.
Back in the console window that you opened, go ahead and run
netstat -a -b -n.Will look kinda like this:
i.sstatic.net/mJali.jpg
That should list all of the programs listening on any ports on the computer. If ollama is up and running on that Windows machine and doing so on the port that I believe it is, then you should have a line that looks like:
TCP 0.0.0.0:11434 0.0.0.0:0 LISTENINGIf it’s 0.0.0.0, then it means that it’s listening on all addresses, which means that any program that can reach it over the network can talk to it (as long as it can get past Windows Firewall). We’re good, then.
Might also be “127.0.0.1”. In that case, it’ll only be listening to connections originating from the local computer. If that’s the case, then it’ll have to be configured to use 0.0.0.0.
I’m gonna stop here until you’ve confirmed that much. If that all works, and you have ollama already listening on the “0.0.0.0” address, then next step is gonna be to check that the firewall is active on the Windows machine, punch a hole in it, and then confirm that ollama is not accessible from the Internet, as you don’t want people using your hardware to do LLM computation; I’ll try and step-by-step that.
BlackSnack@lemmy.zip 18 hours ago
Dope! This is exactly what I needed! I would say that this is a very “hand holding” explanation which is perfect because I’m starting with 0% knowledge in this field! And I learned so much already from this post and your comment!
So here’s where I’m at, -A backend is where all the weird c++ language stuff happens to generate a response from an AI. -a front end is a pretty app or webpage that takes that response and make it more digestible to the user. -agreed. I’ve seen in other posts that exposing a port on windows defender firewall is the easiest (and safest?) way to go for specifically what I’m looking for. I don’t think I need to forward a port as that would be for more remote access. -I went to the whatismyipaddress website. The ipv6 was identical to one of the ones I have. The ipv4 was not identical. (But I don’t think that matters moving forward.) -I did the ipconfig in the command prompt terminal to find the info and my ipv4 is 10.blahblahblah.
I’m ready for the next steps!
tal@lemmy.today 17 hours ago
Yes.
Yes. I’d like to confirm that that is not happening, in fact.
Hmm. Okay, thanks for mentioning the IPv6 thing. It is possible to have the ollama reachable from the Internet via IPv6, if it’s forwarded. I should have thought of that too and mentioned that. Shouldn’t need to open an IPv6 hole in the Windows Firewall, but would rather not rely on the Windows Firewall at all.
It shouldn’t be an issue if ollama is only listening on an IPv4 address. You only see the “0.0.0.0:11434” line, right? No other lines, probably with brackets in the address, that have a “:11434”, right? That could be an IPv6 address.
goes to look for an example of Windows netstat output showing a listening IPv6 socket
Here:
configserverfirewall.com/…/netstat-command-to-che…
Can you just make sure that there’s nothing like
0:[::]:11434in there? That’d be what you’d see if it were listening for IPv6 connections.Sorry, just don’t know oollama’s behavior off the top of my head and want to be sure on this before moving ahead, don’t want to create any security issues.
Yeah, that’s expected and good. The one from the website is your public IP address, anf the one from ipconfig your private one, that you’ll use to talk to the machine wirh your phone.
Great, yeah, that was the right move.
Okay, then just want to sanity check that your iOS device is in the same address range on your WiFi network, that the 10.x.x.x address on your LLM PC isn’t from a VPN or something (since it’s a little unusual to use a 10.x.x.x address on a home broadband router, and I want to make sure that that’s where the address is from). Go ahead and put the iOS device on your WiFi network if you have not already.
This describes how to check the IP address on an iOS device.
servicehub.ucdavis.edu/servicehub?id=ucd_kb_artic…
You should also be seeing a 10.x.x.x address there. If you don’t, then let’s stop and sort that out.
If that’s a 10.x.x.x address as well, then should be good to go.
Oh, one last thing. In the ipconfig output, can you make sure that the “Subnet Mask” reads “255.0.0.0”? If it’s something different, can you provide that? It’ll affect the “/8” thst I’m listing below.
Okay, if you’ve got that set up and there are no other “:11434” lines and the Subnet Mask is “255.0.0.0”, the next is to poke a hole in Windows Firewall on IPv4 TCP port 11434.
kagis for screenshots of someone doing this on Windows 11
windowsreport.com/windows-firewall-allow-ip-range…
I’m assuming that this is Windows 11 on your PC, should have asked.
You’re going to want a new inbound rule, Protocol TCP, Port 11434.
For “local IP addresses”, you want “These IP Addresses”, and enter
10.0.0.0/8. That’ll be every IPv4 address on your Windows LLM that has “10” as its first number — you said that you had a “10.” from ipconfig.For “remote IP addresses”, you want “These IP Addresses”, and enter
10.0.0.0/8. Same thing all addresses that start with a “10.”, which should include your iOS device.Okay. Now you should have a hole in Windows Firewall. Just to confirm that port 11434 isn’t reachable from the Internet, I’m gonna use one of the port-open-testing services online. My first hit is for one that only does IPv4 and another that only does IPv6, but I guess doing two sites is okay. Can you go to this site (or another, if you know of a site that does port testing that your prefer)
www.yougetsignal.com/tools/open-ports/
Plug in your public IPv4 address there (not the private one from ipconfig, the one from that website thst I listed earlier) and port 11434. It should say “closed” or “blocked” or something that isn’t “open”. If it’s “open”, go back and pull thst firewall rule out, because your router is forwarding incoming IPv4 connections to your LLM PC in some way that’s getting, and we gotta work out how to stop that.
port.tools/port-checker-ipv6/
Here’s an IPv6 port tester. Plug in your IPv6 address there (which you said was the same from both the website and ipconfig) and port 11434. It should also say “closed” or “blocked” or similar. If it says “open” — I very much doubt this — then go back and pull out the firewall rule.
If both say “closed”, then go ahead and install Reins.
Based on this:
www.reddit.com/r/ollama/comments/1ijdp1e/reins/
It’ll let you input an “endpoint”.
Plug in the private IPv4 address from your LLM PC, what was in ipconfig, like “10.something.something.something:11434” and you should, hopefully, be able to chat.
BlackSnack@lemmy.zip 2 hours ago
There are 3 lines with the :11434 in them. No brackets or anything like that. -1 has 0.0.0.0 in front -2 has 10.#.#.# in front and has a foreign address that is something other than 0.0.0 -3 is like the 2nd but a slightly different foreign address
The iPhone does have a 10.#.#.# ip number that is slightly different than the PCs.
The subnet mask is 255.255.255.0
I have taken a pause here while we trouble shoot the subnet mask. We’re getting close!!