Laya, JEV and Clef against my rules: a decision benchmark
Three models that pick a label and return probabilities, tested on two real jobs of mine. The small fine-tune won where it was trained, the 27B won where nobody trained anything, and on PR effort plain rules beat them all.
A decision model (a program that chooses between options) takes context (the information it needs) and a list of options. It picks one and returns probabilities (chances in numbers). I use this for two jobs: finding the intent of a spoken phrase for Home Assistant, and estimating the effort of a PR (a request to review code changes). On October 26 and 27, I measured three candidates and a small baseline against what I already had running.
The candidates:
- Laya: my fine-tune (an adjustment made with examples) of laya-multilingual, with 322 million parameters (internal model values). It runs on a Mac with MPS (a feature that speeds up calculations). I wrote about how I trained it.
- JEV: a typed-decision API (a service that takes options and returns a choice) from TypeSafe. It returns a choice, score, or yes/no. It costs $0.04 per million input tokens (units of text sent to the service). It only accepts text and is hosted in the US. Its parameters are not published.
- Clef:
clef:27b, a decision model with 27 billion parameters served through Ollama's/v1/systemoneendpoint. An endpoint is the address used to call a service. I tested the cloud version and the local one. The local version needs an 18 GB download. - tev1:4b: a local model with 4 billion parameters, used as a small reference.
PR effort: the rules win
I used 67 real cases from the shadow log (a test record where the model does not change the decision). Each case produced two decisions, for 134 in total. Rules without a model got 99.3%.
Same 67 real cases, two heads each. Deterministic floors are rules with no model.
Show data
| Deterministic floors | 99.3% |
|---|---|
| Clef (27B, cloud) | 96.3% |
| Laya (typed decisions) | 92.5% |
| Majority class | 66.4% |
| Laya base, no fine-tune | 15.7% |
Clef in the cloud got 96.3%. Its p50 (median time) was 557 ms. Brier at 0.095 and ECE at 0.12 are two measures of probability quality. Laya got 92.5% in 26 ms, but had a Brier of 0.34 and ECE of 0.35. It gets many answers right, but its probabilities do not reflect that certainty well. Its mean maximum probability was 0.58. Base Laya, without the fine-tune, got 15.7%, below even choosing the most common class (66.4%).
Laya local (Ollama, bf16) against Clef in the cloud.
Show data
| Laya | 26 ms |
|---|---|
| Clef (cloud) | 557 ms |
Laya gave the same numbers on Ollama (bf16, a 16-bit number format, 26 ms) and on Python with MPS (30 ms, 2.8 GB RSS, the memory used by the process, and 21 s to load).
To decide where to place each model in the router, I compared 36 synthetic tasks (made for the test). Clef in the cloud and Clef local both got 83% of effort estimates right. They took 580 ms and 3.3 s. JEV got 69% in 1.5 s. tev1:4b got 47% in 350 ms. My conclusion: JEV does not help with routing. Clef local is not worth it either. It had the same accuracy, took about 6 times longer, and uses 18 GB on a machine already under memory pressure.
Clef cloud and local tie at 83%; the difference is latency and memory.
Show data
| Clef (cloud) | 83% |
|---|---|
| Clef (local) | 83% |
| JEV | 69% |
| tev1:4b | 47% |
Home intent: the fine-tune wins where it was trained
Here the test used two sets: 44 probe rows (questions written by hand) plus 258 held-out rows (data kept separate for testing). I measured accuracy, wrong actions among those executed at p >= 0.70 (a probability of at least 0.70), and p50.
- Laya: 98.0%, 4 wrong actions out of 253 executed, 55 ms.
- JEV: 81.5%, 15 out of 184, 1.6 s.
- tev1:4b: 66.9%, 14 out of 133, 4.3 s.
Held-out is Laya's own distribution, so it favors Laya.
Show data
| Laya | 98.0% |
|---|---|
| JEV | 81.5% |
| tev1:4b | 66.9% |
On the 44-row probe, Laya got 41, JEV got 34, and tev1 got 28. The held-out data comes from Laya's own distribution, so it favors Laya. Even so, the gap appears on the hand-written probe.
Ollama's /v1/systemone endpoint limits each choice to 26 candidates. The house has 77
labels (names for the options). So tev1, which uses the same endpoint, needed five calls
to compare them. JEV evaluated all 77 in one call. Clef was left out of this test. On
October 21, I had already decided to keep it out of the house.
What I decided
Laya stays the only decision model in the home assistant. Clef runs in shadow (it logs suggestions without changing the result) for PR effort estimates. The rules stay in charge because they already decide 99.3% of these cases correctly.
The shadow record has to store which model I requested. The response's model field
drops the -cloud suffix. Without the request, you cannot tell cloud from local.