From Sketch to STEP: Designing the Image-to-CAD Pipeline

Turning an image into CAD sounds like it should be a single model doing a single hard thing: look at a picture, output geometry. In practice we split it into two possible approaches with very different cost and risk profiles, deliberately built the cheap one first, and structured the system so the expensive one — if we ever need it — can replace the cheap one without anyone downstream noticing.

Phase 1: a caption bridges the gap

The first approach doesn't train a new geometry-generation model at all. It uses Qwen2-VL-2B-Instruct, a small vision-language model, LoRA-tuned to look at an orthographic-view image of a part and produce a detailed text description in the same style as the prompts the text branch was already trained on — the Text2CAD dataset's "detailed prompt" format. That caption then gets handed straight to the already-working text-to-CadQuery pipeline. We call this the caption-bridge, and its defining property is that it reuses everything: the same text branch, the same executor, the same validator, the same repair loop. The only new component is a captioner sitting in front of a pipeline that already exists.

This is a meaningfully lower-risk build than it might sound. Training a vision-language model to describe a mechanical part accurately is a narrower, better-understood task than training a model to directly emit correct geometry from pixels — and if the captions come out mediocre, the failure mode is graceful: the text branch still works on whatever caption it receives, so image generation degrades to "worse text prompts," not "no output."

Phase 2 exists only if Phase 1 falls short

The second approach is the more ambitious one, and it's explicitly conditional in the design: build it only if Phase 1 underperforms on the held-out evaluation set, particularly on Fusion360 Gallery sketches, which are a harder distribution-shift test than the DeepCAD-sourced training data. Phase 2 skips the captioning step entirely — a frozen vision encoder, either DINOv2-small or SigLIP-base, feeds a small trained projection layer directly into Qwen-Coder's embedding space, with the whole thing fine-tuned end-to-end on (image, CadQuery program) pairs rather than (image, caption) pairs.

The upside of Phase 2 is that it removes a lossy intermediate step — a caption is a compressed, English-language bottleneck between the pixels and the geometry, and compression loses information a direct projection wouldn't have to lose. The cost is that it's a genuinely new model architecture, trained on a new pairing of data, evaluated against its own held-out set, and it only gets built at all if Phase 1's numbers say the captioning approach isn't good enough.

The engineering judgment call

The interesting decision here isn't the modeling choice — it's the sequencing. We could have started with Phase 2, on the theory that it's the "real" solution and Phase 1 is just a stopgap. We didn't, because the two phases share an identical API contract: a client uploads an image, gets a jobId back, and polls or streams status the same way the text branch already works. Nothing about POST /api/generate or the SSE status stream cares which model sits behind the image branch. That means the frontend, the job-state schema in MongoDB, and the Studio UI's image-upload flow can all be built once, against Phase 1, and never touch again if Phase 2 eventually replaces it.

That's the general shape of the trade-off: ship the version that's cheaper to build and lower-risk to operate first, prove out the API surface and the UX against it, and only pay for the more expensive approach if the evaluation numbers actually demand it. A caption-bridge that ships on day one and gets replaced quietly behind a stable API is a better use of a fixed amount of engineering time than a more elegant model that isn't finished yet.