Comment on Is anyone self-hosting their own AI?

• kensand@sopuli.xyz ⁨3⁩ ⁨weeks⁩ ago

  I mean, I haven’t really started discussing it outside IRL friends, so not surprised you haven’t heard of it 😅

  In essence, the problem is twofold:

  1. LLMs tend to use CLIs more effectively than MCP, at least in the domains I use. However, installing all the CLI tools and configuring them can be cumbersome and not easily repeated. CLIs also cannot be modified, and are often quite verbose.
  2. LLMs will repeat very similar commands over and over with just a few tweaks. This often happens when there is a gap in a tool or poor documentation.

  So fcli is intended to be an easily extended, configured, and authenticated CLI. The “one CLI” to do everything your LLM needs. Ideally, you set up a personal repo to store ‘ucks’ (f packages) in, and you set up credentials for that repo so your LLM can push any new ucks to it. You can then share that repo with other users if they could also benefit from any of that functionality.

  As you go along in a session, when you notice the LLM repeatedly running similar verbose commands, you can have it create an uck for it instead and save those tokens.

  Likewise, if, say, an MCP server doesn’t have full functionality, you can have your LLM import the SDK from npm into an uck and write a basic fcli wrapper around it to start using.

  The other nice thing is that it makes your tool configuration repeatable between LLMs and harnesses. You don’t need to set up your MCP servers in both Claude and Codex and Pi, just in f and tell them to use f.

  There’s also a creds uck to make your secrets more portable and (a bit) safer.

  In general, I’m trying to model it a bit like pi.dev, but as part of the tooling instead of the harness itself. I think they work quite well together :)

  Interesting paper comparing MCP and CLI

  original

  • PeriodicallyPedantic@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago

    Can you elaborate on LLMs using CLI more effectively than MCP? Do you have any sense of how likely it is just your domain or the MCPs you’re using? Do you have any theories about why that may be?

    original

    • kensand@sopuli.xyz ⁨2⁩ ⁨weeks⁩ ago

      MCP in general:

      1. MCP is very verbose (json tool declarations)
      2. Every tool incurs a prefix cost, which can be in the thousands of tokens across all tools
      3. Tool outputs are usually dumped straight into the prompt. Sometimes they are truncated and dumped into a file with a warning to the LLM, but usually after they reach thousands of tokens
      4. Changing any tools breaks your prefix cache.
      5. Tool discovery can help with 1 a bit, 2 a lot, and 4 a lot, but isn’t a cure-all. It does have its own (much smaller) token overhead too.

      CLI in comparison:

      1. Fairly concise, by nature - short names for frequently used commands, spaces as implicit delimiters, implied outputs.
      2. Tools are generally implied to be present. A short “this environment contains x,y,z,” system prompt can be very effective. Very minimal prefix cost.
      3. Tool outputs can be written to a file directly, formatted, head/tailed, grepped, or used in any other way.
      4. Doesn’t break the prefix cache, but can “confuse” the LLM.
      5. Tool discovery is implicit in the system. This does have its own overhead, but for many things is implied - ls, find, grep, sed, cat - usually an llm will just assume they’re present, and use them to trim down tool outputs to only the required context. In f, I also added search functionality to avoid parsing ‘help’ messages.

      CLI does of course have its own downsides, I think the biggest being security. Give an LLM an MCP server, and all it can do is what that MCP server supports. Give an LLM a Turing complete shell, and it can do anything.

      MCP also tends to be harder to set up and harder still to fix when it doesn’t work. I’ve watched Claude fall back to curl commands when a connector doesn’t work many times. I’ve never seen it try curl commands, fail, and then try to set up MCP.

      I think this is all fairly domain dependent, but I think my domain (software development and general agenetic assistance) is very common. If you’re using an LLM in a product that you’re selling with a fixed range of features, MCP is probably the way to go. If you’re using an LLM for dynamic functionality with expanding capabilities, I think CLI might fare better.

      My main reasoning for all of this is flexibility. In the history of computing, a program has always had a fixed output for a fixed input. LLMs are at the point where they can handle what is effectively an arbitrary input and give an arbitrary output. Giving an LLM a fixed set of tools negates that newfound ability to process the unknown. It’s like giving someone a sledgehammer to bust rocks, but tiring one hand behind their back; the work might get done, but not as efficiently.

      original

      • PeriodicallyPedantic@lemmy.ca ⁨2⁩ ⁨weeks⁩ ago

        I agree with you for common CLIs that the model will natural have in its training set, but when building your own CLI it seems like the self-documenting nature and progressive disclosure of MCP would have a huge advantage.
        With a cli you don’t have the prefix cost, but your agent also doesn’t know that the tool exists, what it’s for, how to use it, etc. once you give it that information, then you have the same issues with prefix cost, right? Changing tools does break the prefix cache, but changing tools is a thing you can do. You can’t change (model awareness of) CLI, and if you could then it’d also break the prefix cache, or you’d have tool instructions scattered throughout the context without structure.
        Dumping to file is a harness thing, not a MCP thing. A harness will do the same with large terminal outputs.
        Likewise setting up a new CLI can be complicated. I never see an agent try to install an mcp, but I also don’t typically see it try to install a new CLI.

        I don’t see how mcp is any more a fixed set of tools than CLI is. It’s just presented to the agent in a structured way, which is important for discovery and instruction following.

        But yeah ease of chaining is certainly useful wrt CLI. That’s a huge win for sure.

        original