Author
Convai Innovations (GitHub)
1 reading card from 1 book · 2026.
1 card
Laya — README-ul proiectului, secțiunile «Benchmarks» și «Honest limits» · 2026
Measure ECE on held-out data and refit the temperature before you put a threshold on a probability.
Speed, with its source. Jev, measured independently (AbdelStark, nibzard, September 2026): 236–276 ms p50 for one question, 70–500 ms stated by the company. Laya on a T4: 32.8 ms multilingual, 39.5 ms English, and in batches of ten 7.2 ms per question — 103–332 questions per second on a single GPU. A frontier LLM on the same judgement: 8.83 s median (Every, 2026). Cost: $0.042 per million input tokens at Jev; $0 per token at Laya plus the GPU and someone's hours. The 7.8× ratio between the two is not a property of the architecture but of distance: one is a network call, the other a local pass. Calibration is the figure that matters. ECE — expected calibration error — compares declared confidence with actual accuracy per bin: an ECE of 0.1 means that, on average, the model's "0.8" is true 70% or 90% of the time, not 80%. Laya ships over-confident (0.466) and reaches 0.081 after you refit one temperature per question type and option count; Jev is reported at 0.144–0.246 without adjustment. The temperature is a single scalar that divides the logits before the softmax: above 1 it flattens, below 1 it sharpens. It does not change which option wins, only how sure the model claims to be — exactly what your thresholds read. When neither helps: arithmetic, date comparison, multi-hop questions, large noisy states, adversarial text — the TypeSafe documentation lists them all. And a warning from the Laya benchmark: on Khmer, the English checkpoint gives 0.000 accuracy at 95.2% confidence. Confidence gives no warning about what it never saw in training; that you verify yourself, on held-out data, before any threshold.
“Both checkpoints are over-confident as shipped. Refitting one temperature per (question type, option count) on held-out data moves mean ECE 0.466 -> 0.081”