Which model speaks URML best, and when a robot should ask

Two additions to URML’s LLM bridge shipped this week. One measures models; the other gives them a third honest move.

urml bench: the numbers behind “works with any model”

URML has said from the start that the language layer is provider-neutral: Anthropic, OpenAI, and local open-weights models through Ollama, llama.cpp, vLLM, or LM Studio are all first-class. What we could not tell you until now is how well any particular model actually speaks URML.

urml bench answers that. It runs a corpus of natural-language requests through exactly the loop urml translate uses (same prompt contract, same JSON repair, same revision budget) against one capability manifest, and reports where each request landed:

• accepted: the validator accepted a program that does real work
• honest refusal: the validator accepted a program whose root behavior is only report(status: failure); the model declined
• invalid emission: the revision budget ran out
• provider error: the output was not JSON even after conservative repair
• policy block: only hardware-provenance errors remained

Each corpus row also declares what a good model should have done, so an honest refusal of a doable request shows up as a miss rather than being binned with invalid output. One run measures one model; urml bench --render aggregates runs into a comparison table.

Two honesty rules before anyone quotes numbers. Refusal means report-only at the root: a program that moves the robot and then reports failure did real work and counts as accepted. And the benchmark measures admissibility against declared limits, not physics: whether an accepted program achieves the user’s goal on a real robot is out of scope, the same line the validator itself draws.

The starter corpora are small (a home robot and a cobot cell, ten and eight requests). They will grow, and every number we publish will come from a committed result row with the model, date, and corpus named. If you run a local model, one command gives you your own row:

pip install urml-validator "urml-llm-bridge[ollama]"
urml bench --corpus bench/corpora/home-en.yaml \
  --manifest reference/validator/tests/fixtures/manifests/turtlebot4_home.yaml \
  --provider ollama --model <your model> --no-policy

Clarify mode: refuse or ask, never guess

The second addition starts from a gap the validator cannot close alone. URML’s posture has two outcomes: a request the robot’s manifest cannot satisfy is refused with a machine-readable reason, and a request it can satisfy becomes a validated program. But there is a third case: a request the manifest can satisfy more than one way. “Bring me the mug” when the room holds two. Today the prompt contract tells the model to pick a reasonable manifest-grounded default, which is a polite name for guessing, and a guessed program is admissible, so the validator rightly accepts it. Guessing is the one failure validation cannot catch.

Clarify mode (RFC-0700, a Draft under review) adds the honest middle path. When the operator enables it, the model’s first emission may be, instead of a program:

{"clarify": {"question": "Which mug, the red one or the blue one?",
             "options": ["red", "blue"]}}

The CLI relays the question, reads your answer, folds it into the request, and translation continues through the ordinary validate-and-revise loop. One question per translation, only before any rejection, and never about something the manifest already decides. The budget is enforced below the model: while it lasts, schema- and grammar-constrained providers (Ollama structured output, llama.cpp GBNF) decode against a program-or-clarify union, and once it is spent the constraint narrows back so the model physically cannot ask again.

The mode is off by default, and off means off: the prompt, the provider calls, and the outcomes are byte-identical to the previous contract, which is what lets the code ship while the RFC is still a Draft. Turn it on with --clarify on urml translate or urml run.

Both features are Apache 2.0 like everything else here, hermetic to test (no network, no API key), and in main now.