Nothing in the support-triage example claims to be validated against the live model. That is by design: it is a placeholder for work that has not happened yet. Here is the plan I would actually run before trusting any of it with real tickets.
Write down the decision and its cost
Evaluation has to start from what a wrong answer costs, not from a generic accuracy number. In the triage example, missing a genuinely urgent ticket and flagging a routine one as urgent do not cost the same thing. One delays a customer who needed help now. The other wastes a few minutes of an agent’s attention. Getting the team routing wrong costs a re-route, mildly annoying but cheap to fix. Getting the impact score badly wrong could mean under-prioritizing a customer who is actually blocked.
Write these costs down per branch before measuring anything. One uniform decision rule applied across all three judgments assumes they are equally risky, and they are not.
Build a useful evaluation set
A representative set means real, messy tickets. The clean examples used earlier to illustrate the API shape do not count. It needs cases that are genuinely ambiguous between teams, tickets with missing context, messages that state urgency the customer does not actually have, requests that try to talk their way into a higher priority, and, importantly, tickets that are genuinely and unambiguously urgent, so you can check the model does not wave those through to routine either.
Reserve a held-out slice you do not touch while tuning thresholds, the same discipline any classifier needs. Treat language as its own variable: Jev’s primary training language is English, and TypeSafe’s own documentation is explicit that other languages, Spanish included, are handled but with lower accuracy. An English evaluation set tells you nothing reliable about performance on Spanish tickets. If this site’s Spanish readers are building the same kind of workflow, that is a separate evaluation, never an assumption carried over from English results.
Measure the whole workflow
Accuracy on the suggestions the system surfaces is not a useful number by itself. Every result the code returns, including a suggestion, still carries requiresHumanReview: true, so nothing here is auto-eligible, and tightening a threshold does not change that. What tightening a threshold actually moves is the split between three numbers, measured together for each candidate policy: suggestion coverage, the fraction of eligible tickets that come back as kind: "suggestion"; defer rate, the fraction that come back as kind: "review"; and suggestion error rate, using only the suggested cases as its denominator. Report invalid answers and service failures separately too, so they do not quietly disappear from the coverage number. Whether a given coverage level actually saves reviewer time is a different question, one that needs its own measurement of how long a review takes with and without a suggestion attached. It does not follow from coverage alone.
Pin the model version while measuring. On 21 September 2026, jev-1.13.0 is what jev-latest and jev-preview both point to, but an alias can move to a new release between tuning a threshold and checking it again, and a new version is not obligated to behave identically at the same confidence value. Log the version every real response reports back, and re-measure before trusting an old threshold against a new model.
None of this has been run yet for this project. This is the protocol. I do not have results to report yet.
Decide what can run automatically
TypeSafe’s own guidance on confidence describes three rough bands: high confidence where you act automatically, medium where you proceed with caution, low where you route to a person and do not guess. That maps directly onto what changes once actual evaluation numbers exist. Right now, every path through the triage example ends in requiresHumanReview: true, on purpose, because no policy has been measured yet.
Once a specific branch has been evaluated, low-stakes team routing at a measured confidence floor, for instance, and the measured mistake rate at that floor is acceptable for what a wrong route actually costs, that branch can graduate to acting automatically, and only that branch. Urgency detection is different: given the uneven cost of missing something genuinely urgent, it would need much stronger measured evidence, a policy proven on far more held-out cases, before I let it skip a person entirely, if I ever would. That is not the same as simply raising urgentYesMin; nudging the raw probability cutoff up does not by itself reduce missed urgent cases, and could make them worse. There is no single safe number that applies everywhere. The number has to come from measuring your own cases against what an error there actually costs you.