Comment on Open-Source, Language-Agnostic Mutation Testing Tool Using LLM Agents
dandi8@fedia.io 1 week ago
On the one hand, mutation testing is an important concept that more people should know about and use.
On the other, I fail to see how AI is helpful here, as mutation testing is an issue completely solvable by algorithms.
The need to use external LLMs like OpenAI is also a big no from me.
I think I'll stick to Pitest for my Java code.
coderinsan@programming.dev 1 week ago
So mutation testing is able to create mutations that are not created by traditional mutations, these are mutations that are more dependent on contextual understanding of code which LLMs excel at. We do preprocessing on our side where we generate a minimal AST of all covered files and pass it to give the LLM a rich contextual understanding of the codebase, allowing us to generate good mutations. Also we make use of LiteLlm so it completely works with open source models too.
coderinsan@programming.dev 1 week ago
Here is a YouTube video with a more in depth explanation of the benefits of using LLMs for mutation testing www.youtube.com/watch?v=8h4zpeK6LOA