Evaluation report
A measured answer, from nine experiments and twenty-eight predictions fixed in writing before any data was collected.
The findings are at the end. If you want the results before the method, go to Conclusions, which collects every prediction, every surprise, and what the results change. The sections between here and there define the terms those findings use.
This reports an evaluation of jev, a decision model sold by TypeSafe. The evaluation followed a plan written before any request was sent. That plan fixed the nine experiments, the sample size of each, and twenty-eight predictions, each stated with the result that would prove it wrong. Nothing here was chosen after seeing an outcome.
The purpose was not to grade the model. It was to find out which of several proposed ways of using it would work. Each experiment therefore ends with what it settles, and the Conclusions gather those into three decisions a reader can act on.
One run produced everything below: 123,805 requests over 138 minutes, at a cost of $12.69. Five requests failed and were excluded; they are counted, never replaced by a default value.
jev answers questions about a piece of text. One request carries a single state — the text, or a structured object — and one or more questions about it. Every question in a request is answered from that same state, and a request may carry many questions at once. Whether asking many at once costs anything is the subject of the third experiment below.
Questions take three forms. This document uses the vendor's names for them.
A noul returns no confidence. Only a choice and a score do. That absence matters in the ninth experiment, which asks whether confidence drops when the model cannot know the answer; there, a substitute had to be used for yes-or-no questions, and the substitute is named where it appears.
Every request in this run was answered by version jev-1.13.0.
A model that reports a probability is making a claim that can be checked. If it says 0.9 on a thousand questions, about nine hundred should turn out true. A model whose probabilities behave that way is called calibrated.
Calibration is the stated reason to choose jev over a language model, and a vendor's calibration on the vendor's data does not establish calibration on yours. So the evaluation asks three questions:
Four rules applied throughout.
The plan set five grades and applied them to every condition separately, not to an experiment as a whole. A model that does well on easy problems and fails on hard ones is useful if you know where the change happens, so the tables below report a grade for each difficulty rather than one grade overall.
Every formal subject here — satisfiability, reachability, Sudoku — lies outside what a model trained on decisions was built for. A poor result on one could mean the subject is hard, the model is weak, or the measuring apparatus is broken. Two controls separate those.
The positive control is a set of support-ticket routing problems with unambiguous answers, run alongside every experiment. It scored 0.980 throughout this run. A weak result on formulas can therefore be read as a fact about formulas.
The smallest difference worth reporting comes from the first experiment below, which measures how much the model's own answers vary between identical requests. That variation is 0.0112. Differences smaller than about twice that figure are not findings, and no claim below rests on one.
The sections below quote the experiments' own wording: the reasons the grading rubric generates, the outcome text each experiment writes, and the predictions the plan fixed in advance. Most of the terms there are standard usage in some field — monotone and total variation from mathematics, witness from complexity theory, noise floor from signal processing, phase transition from the satisfiability literature, ceiling from experimental psychology, and the named statistical tests from statistics. Two are particular to this plan and would not be recognised elsewhere: enrollment and signature. All are defined here because a term being standard in its own field is no help to a reader who works in another one.
Each experiment reports one row per difficulty. Deciding metric names the measurement that set the grade for that row, value is what it came to, baseline is what it had to beat and where that number came from, and why gives the rule that produced the grade. Where a grade was set on a smaller sample than the row's own, both counts are shown.
In the order the plan ran them, which is the order in which an early result could invalidate a later one. Each gives the predictions made in advance, the method, the result, what was unexpected, and what it settles.
The plan numbered these E1 to E9 in a different order, and those labels survive in the file names of the recorded results. Both numbers appear on each heading below: the position in the run, and the plan's own label.
Twelve rules that follow from the results above, in four groups. Each gives the requests to write, the mistakes to avoid, and the measurement behind the advice.
Figures come from one of two places. Those attributed to an experiment are
from the full run against jev-1.13.0, and the sample size is given with each.
Those described as measured live were run while this page was written, on sixty problems
per condition. Both are reproducible from the seeds in the repository.
A difference is treated as real here only if it exceeds the model's own variation between identical requests, which the first experiment puts at 0.011. Anything smaller than roughly twice that is not a finding, and is not used as one below.
Every prediction, scored against the condition the plan set for it in advance; then the behaviours no experiment was designed to find; then the three decisions this run settles.
The plan noted that a high proportion correct would be mildly suspicious, since it would suggest the experiments had been set too easily. Predictions that turned out wrong are the most useful part of this section and are listed first.
Before any data, the plan named one result it expected to be the biggest surprise and one it expected to matter most. They came out differently.
That asking many questions at once would turn out to degrade the later ones, showing the parallel evaluation to be less complete than advertised.
It does not degrade them. Accuracy at question 200 matched accuracy at question 1 to within the measurement. The architecture behaves as described.
That if confidence fails to track ignorance, the recommended way of deploying the model — act when confident, escalate when not — stops working, however good the accuracy figures look.
By the measure the plan named this holds: the mean separation is 0.199, below the 0.30 set for a usable signal. But a mean is the wrong statistic for a quantity that sits at exactly 1.00 on 86% of answerable states. Ranked instead, confidence on a choice question separates every kind of unanswerable state, at 0.893 to 0.992. The deployment pattern works — at two gates rather than one, and only if you ask as a choice. The abstention section above has the detail.
Three patterns appeared across every experiment that measured them.
Probabilities are quantised to two decimal places. All 3,194,813 probabilities seen in the batching experiment fall exactly on a hundredths grid. No calibration measurement here can resolve anything finer than 0.01.
Probabilities cluster rather than spread. The value carrying the decision was exactly 1.0 on 52 percent of 836,700 answers in that same experiment.
Confidence is not the largest probability. The two differ on 47.1 percent of choice and score answers, by 0.09 on average and by as much as 0.37. The vendor documents neither quantity's definition, and the difference is large enough to be a usable signal in its own right.