B.E.T.H. · Research notes
Research notes
B.E.T.H. is the test bench for ongoing doctoral work on whether a small language model running locally on consumer hardware can reliably map plain spoken language onto concrete application actions. Every figure below is repeated across trials, because single runs turned out to be unstable enough that one pass is not a measurement, it is an anecdote with a decimal point on it.
The results are also split by OS build, which was not the original plan. Two builds of the same Apple on‑device model behave differently enough that a number without a build attached to it does not mean anything. That distinction is the most useful thing this work has produced so far, so it comes first rather than in a footnote.
The evaluation was run first against the macOS 27 beta, and then, after rolling back, against macOS 26.6.2, which is the build most people are actually on. The failure modes are not the same. Given a sentence with nothing to extract, the release build tends to fill the empty field with a plausible invented noun rather than leaving it empty; the beta was more willing to return nothing. That difference matters more than any accuracy figure, because everything downstream was built on the assumption that empty means empty.
Worth noting which direction that runs. The newer beta was the better-behaved one, and the build in front of real users is the one that invents. Whatever produced that is not something a developer can wait out or plan around, which is the practical argument for not letting a model's output reach anything consequential unchecked.
The validator that lifted intent accuracy was signature‑based: it checked the shape of what came back and rejected anything that did not fit. On macOS 27 that caught most of the errors. On 26.6.2 it caught far fewer, because a confidently invented noun has exactly the same shape as a correct one. There is nothing structurally wrong to detect. Grounding checks, meaning testing whether the extracted term actually appears in what the user said, catch a good deal of it, but they cannot tell a confabulation apart from a legitimate paraphrase, and a user who says "the thing I just opened" is paraphrasing on purpose.
On macOS 27, the model identified the correct intent in about 78% of cases unaided. Placing a deterministic validator between the model and the action — ordinary code that checks the proposed intent against what the app can actually do, and rejects it otherwise — raised that to roughly 94%. Reliability came from the code around the model, not from the model. Read that pair as a result about one build, not about the framework: the same validator does not reach the same figure on 26.6.2, for the reason described above.
How the output structure is defined changes what the model produces. Adding a self-assessment field to the schema, asking the model to report its own certainty alongside the answer, suppressed extraction of the answer itself. Constrained decoding propagates: a field added at the end can quietly degrade the fields before it.
Mean reported confidence was 0.67 when the model was right and 0.63 when it was wrong. That gap is too small to act on, which rules out the obvious hybrid design of routing low-confidence requests to a cloud model. Whatever decides when local is not good enough, it cannot be the model's own estimate of how it did.
The model does not reliably distinguish a referring expression from its referent. Asked to act on the thing a phrase points at, it will act on the phrase. This is not a prompting problem to be tuned away; it looks structural, and it is where the dissertation work is now aimed.
About 2.6 seconds from launch to first spoken reply, once settled. Fast enough to be usable, slow enough that the interface has to say something while it waits.
Mostly it moves the burden onto measurement itself. Any claim here has to carry a build identifier and a trial count or it is not a claim, and a result that appears once and cannot be reproduced across runs gets thrown out rather than reported. The practical consequence for anything that ships is that a model‑based component has to beat a deterministic baseline written first, on the same test set, before it earns the right to be in the product.
Work in progress toward a dissertation defense in 2028. Figures are from a pilot at small scale, on specific OS builds, and should be read as direction rather than as final results. Where a figure is not attached to a build, assume it is not yet trustworthy.