I understand LLaMA and some other models come with instructions that say that they cannot be used commercially. But, unless the creators can show that you have formally accepted a license agreement to that effect, on what legal grounds can that be enforceable?

If we look at the direction US law is moving, it seems the current legal theory is that AI generated works fall in the public domain. That means restricting their use commercially should be impossible regardless of other circumstances - public domain means that anyone can use them for anything. (But it also means that your commercial use isn’t protected from others likewise using the exact same output).

If we instead look at what possible legal grounds restrictions on the output of these models could be based on: if the creators of these models can show that you agreed to a license agreement they may be able to that. However, if you didn’t agree to a license agreement, I don’t see on what basis any restriction in the use of AI model output can be based on. Copyright don’t restrict use, it restricts redistribution. The creators of LLMs cannot reasonably take the position that output created from their models is a derivative work of the model, when their model itself is created from copyrighted works, many of which they have no right to redistribute. The whole basis of LLMs rest on that “training data” -> “model” produces a model that isn’t encumbered by the copyright of the training data. How can one take that position and simultaneously belive “model” -> “inferred output” produces copyright encumbered output? That would be a fundamentally inconsistent view.

(Note: the above is not legal advice, only free-form discussion.)