Apple researchers find sketching and direct manipulation better for training UI models

Apple's researchers set out to fix a specific problem: LLMs generate bad UIs, and the usual RLHF fix of thumbs-up rankings doesn't capture why a designer dislikes something. They ran a study with 21 designers producing roughly 1,500 annotations via commenting, sketching, and direct manipulation, then finetuned models on that data. Judged by humans, the resulting models beat both ranking-trained baselines and GPT-5.