Small Models, Hard Decisions

Laya led the first test

I started with 72 authored cases. Fifty-four had a definite answer, 12 were missing evidence or outside the requested domain, and six described futures that could not yet be known. Neither model had trained on these cases.

Laya answered 47 of the 54 definite cases correctly, or 87.0%. Verdict answered 35, or 64.8%. Laya was especially stronger on yes/no and scoring questions, while Verdict had a slight edge when choosing from a list.

The tradeoff was size. Verdict's complete download was about 309 MB; Laya's was about 847 MB. Verdict could also decline four of the 12 missing-evidence or out-of-domain requests, while Laya had no native abstention outcome and answered all 12. Accuracy and restraint turned out to be separate problems.

Then I ran the converted models through Core ML on a Mac. Laya kept all 72 decisions from the original model. Verdict changed two. A conversion that loads successfully can still alter the product's behavior, so I made decision parity a separate gate.

These were Mac measurements. The compute-unit request left actual execution placement uncertain. iPhone latency, memory use, and energy consumption still require device tests.

Agreement exposed shared mistakes

The next idea was simple: use Laya and Verdict as teachers, keep the decisions where they agreed, and train one smaller student to reproduce that combined signal. If disagreement filtered mistakes, perhaps one model could learn the safer subset.

The first student experiment used synthetic rule-based numeric questions. Four students started from the same Verdict weights: one learned from independently generated labels, one from Laya, and two from hard or soft agreement between Laya and Verdict. At the default acceptance threshold, all three teacher-supervised students declined every validation case. The independently labelled student answered the definite cases, but was only 64.7% correct. The training code ran successfully and gave me no evidence that teacher supervision helped.

I then removed most of the arithmetic and built a broader semantic set around support requests, workflow states, deliveries, document status, and entry policies. Before spending time on another student, I required the two teachers to pass a set of quality gates together.

The two models agreed 235 times. Only 165 of those agreements were correct. Thirty-nine were shared wrong answers, and 31 answered questions that should have remained unresolved. On 14 accepted false yes/no propositions, the pair was wrong all 14 times. They also made the same mistake of following an archived status note instead of a newer, authoritative report.

The denominator mattered, too. On the first diagnostic, the agreement rule got 30 of 31 accepted definite answers right: 96.8%. But it also answered six cases with missing evidence or unknowable futures. Counting every answer the user would actually receive changed the result to 30 of 37, or 81.1%.

All returned answersDefinite cases only

Definite cases only

Agreement can filter disagreements. It cannot certify a shared mistake.

MiniLM was smaller and less accurate

I also tested an unchanged 82-million-parameter MiniLM model. It was less than a fifth of Laya's parameter count and seemed like a more realistic starting point for an on-device feature.

On 90 short relation probes, MiniLM scored 83.3%. When I added repeated but explicitly unrelated administrative text, accuracy fell to 33.3%. Every padded example became neutral, and 54 of the 90 decisions changed. The result captures sensitivity to this type of synthetic padding; short, tidy sentences had hidden it.

I then mapped MiniLM's entailment, contradiction, and neutral outputs into the same yes/no, choice, and score interface. Two reasonable prompt formats both missed the quality, coverage, and unknown-rejection targets.

Core ML introduced one more tradeoff. The roughly 165 MB FP16 packages produced invalid values or changed decisions. The roughly 330 MB FP32 packages preserved the original decisions and probabilities on the Mac. The larger conversion faithfully preserved the model's weak behavior.

Building the Swift interface

Alongside the model experiments, I built a Swift interface for yes/no, choice, and score questions. It supports batch evaluation, editable rubrics, visible probability distributions, and an explicit unknown state.

• Unknown has its own value. It stays separate from false, a score of zero, and a low-confidence guess.
• Swift handles deterministic rules. Dates, arithmetic, missing required fields, and workflow transitions stay in code.
• Core ML output is checked against the source model. The app can reject a converted package when its decisions change.
• Accuracy is reported with coverage. The results show how often a model answers as well as how often it is correct.

Twenty-three Swift tests passed, and the unsigned iOS release build compiled. Laya remained the default development backend because it led the first test and preserved its decisions after Core ML conversion.

I ended the experiment there. The models had missed the broader quality thresholds, so I did not proceed to signed installation, offline relaunch, or phone performance testing. Using existing models shortened the implementation work. None of the evaluated candidates met the quality bar for deployment.