Bunch of reasons, few people have pointed out in other comments already, including:

• text =/= dialogue
• LLMs train on the average guy, not good actors or only well done plays
• AI models are statistical approximations of patterns that emulate behavior of the real world. It’s gonna take shortcuts wherever it can
• maybe there’s specific tricks for positive or systems prompts that will improve quality in your case / on you model
• the model you are using could be bad for the task
• some of the points you are making strike me as personal preference; they could just as well apply to non-AI written series

Those are some things that play into why it feels like it’s awful at it. Some of it perception, some of it is true.