The Repair Loop: Teaching a Model to Fix Its Own Mistakes

A model that writes code will sometimes write code that doesn't run. That's true of a large frontier model with a hundred billion parameters and it's true of a QLoRA-tuned 1.5B-parameter Qwen2.5-Coder checkpoint, which is what powers SolidMake's text branch. The naive response to a failed generation is to fail the job and show the user an error. We do something else: we treat the traceback as information the model hasn't seen yet, and we give it a second chance to use that information.

What actually happens on a failure

Every generation runs through a sandboxed executor before anything is shown to the user — the model's output is a CadQuery Python program, and CadQuery programs, like all Python programs, throw exceptions. A profile that doesn't close, a boolean cut on a solid that doesn't exist yet, a fillet radius larger than the edge it's applied to — these are ordinary bugs, not signs of a broken model.

When the executor raises, the pipeline packages three things together: the original prompt, the code the model just generated, and the traceback itself. That bundle goes back to the model with an explicit instruction to fix the error, and the model re-samples. This is the repair loop, and per the generation-pipeline design it runs for up to two retries before a job is finally marked failed. The executor's timeout — 60 seconds wall-clock, enforced by a SIGKILL — counts toward that same retry budget, so a hang is treated the same as a traceback rather than as a separate failure mode.

Why the traceback is worth feeding back

The reason this works at all is that CadQuery is strict. Python's parser and OpenCascade's geometry kernel both fail loudly and specifically — a KeyError on an undefined variable, a BRepBuilderAPI error on a self-intersecting wire — rather than silently producing garbage. That strictness, which we picked CadQuery over raw DeepCAD operation JSON for in part because of it, is what makes the repair loop viable. A vague failure gives the model nothing to correct; a Python traceback with a line number gives it a specific, addressable mistake.

This isn't a novel idea we invented — the Text2CAD paper's own ablations found that feeding a single execution error back to the model for one re-attempt recovers on the order of 40% of initially failing generations. That's not a number we're claiming for our own fine-tuned checkpoint yet since evaluation happens after training, but it's the empirical basis for why the design allocates retry budget to this instead of, say, just re-sampling from scratch with a different temperature.

Why this matters more than it sounds like it should

Consider two versions of the same underlying model: one wrapped in a pipeline with no repair loop, one wrapped in a pipeline with a two-retry repair loop. If the base execution rate is 65%, the first pipeline fails on roughly a third of all prompts, full stop. The second pipeline, recovering something like 40% of those failures on the first retry, and a further slice on the second, can land close to 90% without the underlying model getting a single parameter better.

From the user's seat, that's the entire product. Nobody grading a demo cares whether the intelligence lives in the model weights or in the scaffolding around them — they care whether the part came out. This is also why the repair loop is a job-lifecycle state (repairing) rather than something hidden inside a single request: the Studio UI surfaces it, so a user watching their generation stream sees the pipeline notice its own mistake and try again, rather than experiencing an unexplained delay. Reliability, in other words, isn't purely a model-quality problem. Sometimes it's an architecture problem, and the fix is deciding that a traceback is a second chance rather than a dead end.