Project-reported headline results
| Dataset | Laya result | Jev published result | Important context |
|---|---|---|---|
| typed-decisions | 0.766 | 0.727 | Laya’s workflow-specific fine-tuned checkpoint |
| AG News | 0.950 | 0.910 | AG News was in Laya’s training mix |
| DAIR Emotion | 0.595 | 0.480 | Held out for Laya |
| Banking77 | 0.425 on 77 labels | 0.870 on 72 labels | Clear high-cardinality weakness at default budget |
These numbers are not one controlled head-to-head run. The Laya project explicitly says it had no TypeSafe API access and that prompts and sample sizes may differ. Treat the comparison as directional.
The checkpoint matters more than the brand name
| Model on typed-decisions | Accuracy | Soft accuracy | Brier ↓ | ECE ↓ |
|---|---|---|---|---|
laya-typed-decisions | 0.766 | 0.471 | 0.062 | 0.213 |
laya | 0.362 | 0.332 | 0.316 | 0.175 |
laya-multilingual | 0.342 | 0.326 | 0.439 | 0.285 |
| Jev 1.13.0, published | 0.727 | 0.580 | 0.148 | 0.144 |
| Majority-class baseline | 0.461 | — | — | — |
The base Laya checkpoints are below the majority-class baseline here. The impressive result belongs to specialization. Jev better matches the teacher’s full probability distribution and has better unadjusted ECE in this table.
Latency on a Tesla T4
| Questions in one call | English Laya | Laya multilingual |
|---|---|---|
| 1 | 39.5ms | 32.8ms |
| 5 | 84.5ms | 40.1ms |
| 10 | 158.6ms | 72.3ms |
| 50 | 771.3ms | 337.4ms |
These are warm local GPU timings. A production router can take roughly 7–10 seconds when it must swap checkpoints, according to the repository, so servers handling multiple languages should preload the models they serve.
Multilingual evidence
On MASSIVE intent classification with 20 options, the multilingual checkpoint reaches 0.366 macro accuracy across 51 languages and clears the project’s “three times random” usability bar in 45 of them. The English model reaches 0.227 macro accuracy and clears that bar in 23.
“Supports 100+ languages” describes the encoder’s intended coverage, not verified performance in 100 production settings. Public evaluation covers 51 languages, and results vary widely. Routing is part of the product, not an optional optimization.
Application themes
| Theme | Best reported checkpoint result | Training status / caution |
|---|---|---|
| Email spam | 0.993 | In training mix |
| Phishing | 0.993 | In training mix |
| LLM guardrails | 0.762 | Held out |
| Moderation | 0.530 | Held out; weak macro-F1 |
| RAG relevance | 0.657 | In training mix |
| Support triage, 10-way | 0.522 | In training mix |
| Model routing | 0.659 | Held out; English checkpoint |
A polished demo can make moderation look solved, while the held-out result says otherwise. Production evaluation should include natural class imbalance, ambiguous cases and adversarial inputs—not only clear examples.
Calibration repair
The repository reports mean ECE changing from 0.466 to 0.081 for the English checkpoint and from 0.314 to 0.106 for multilingual after fitting temperatures by question type and option count. Those post-fit values are encouraging, but calibration fitted on one distribution can drift on another.
Independent evidence: useful, but small
A separate M4 Max test compared multilingual Laya-MLX with Jev 1.13 on 40 Chinese support-ticket cases. Jev scored 31/40; Laya scored 23/40. Laya’s warm local latency was 7.6ms versus a 588ms Jev API call. At a selected cascade threshold, the test matched Jev accuracy while sending 55% of cases to Jev.
The author clearly notes the limits: one task, one machine, one small test set and arguable labels. It should generate hypotheses, not headline claims.
A responsible evaluation checklist
- Freeze a labelled set before tuning prompts, criteria or thresholds.
- Report the exact checkpoint, package version, hardware and context limits.
- Measure macro-F1 and per-class recall alongside aggregate accuracy.
- Plot reliability by confidence bucket and inspect confident errors.
- Shuffle option order and test paraphrases.
- Include ambiguous, empty and out-of-scope inputs.
- Separate warm inference from model loading and end-to-end service latency.
Main data source: Laya BENCHMARKS.md and linked raw result files. Independent sample: laya-jev-lab. Values were transcribed on 22 September 2026 and may change.