Discussion about this post

User's avatar
Glen's avatar

Very interesting read. I wonder if the llm is getting tripped up by the system prompt statement to not refactor code. Modifying failing code could be considered refactoring to it since it lacks understanding of the code (it only mimics understanding…) I wonder if an extended thinking model may thrive because of this.

I wonder how the system performance might change by modifying the restrictive “begging” to avoid refactoring, and if it would allow the usage of a simpler/cheaper model and also would it have helped the failed implementations .

Expand full comment
Steffen Börner's avatar

this is great! i made the experience that, in many cases, AI simply ignored my instructions to use a TDD cycle. But i found a way to let AI and me stay in the TDD loop together:

I once wrote an extension to visualize the current TDD phase, and with a command, actively switch to the next phase (VSCode Marketplace: tdd-helper). This would help me to stay in the tdd cycle while developing. Turns out, this extension now is helping AI as well: i trigger the next phase, the extension writes the updated TDD phase to a json file, where AI will read the phase before doing anything. This way, AI will reliably do what guidelines i gave for the specific phase and i am in control when to switch to the next phase, having the time to review what it implemented.

Expand full comment
9 more comments...

No posts