Six real bioinformatics and research-tooling tasks, one new kind of AI model, one cheap open-weight LLM tuned for each task, and about $4 of API spend. Plain language, real numbers.
The AI models most people use (ChatGPT, Claude, DeepSeek) generate text one word at a time. Ask them to sort 10,000 records into categories and you get 10,000 little essays, which your code then has to parse, check and occasionally repair.
Jev, from TypeSafe AI, does something narrower. You give it a block of text and a set of typed questions: pick one of these options, rate this on a scale, or is this statement true. It returns a probability for each answer in about a quarter of a second. It cannot answer with anything outside the options you gave it, so there is nothing to parse. Its makers call this a "System One" model, after Kahneman's fast, intuitive mode of thinking.
| Type | You ask | You get back |
|---|---|---|
| Choice | pick one of a defined set (up to 255 options) | the pick, a probability for every option, and a separate confidence |
| Score | rate against ordered levels (low, medium, high) | the level and a probability per level |
| Noul | is this statement true? | a probability that it is |
Every question in a request is answered at once, so asking twenty questions costs about the same time as asking one. Price is $0.042 per million words of input; the answers themselves are free.
Public genomics repositories hold hundreds of thousands of experiments, each described by whoever uploaded it, in their own words: "HepG2 H3K27ac rep1", "iPSC-derived neurons at DIV16", "WAT MSC". The task is to read that free text and pick the tissue class (20 options) and the assay type (7 options).
Reusing public data means harmonising it first, and curators have been assigning these categories by hand for years. It is the highest-volume, lowest-glamour job in the field, and it is exactly the shape Jev is priced for: short text in, one of a fixed set out, at scale.
5,000 records sampled from ChIP-Atlas, stratified by class, with the curators' choices as truth. 500 for tuning the wording, the rest scored once. Every model got the same records and the same questions. The no-model baseline is always guessing the commonest class. A second, untouched 2,999-row sample scored the tuned pipeline.
* best on this metric. (tie): the accuracy difference is within noise on a paired test.
Both right. The difference on this record is only the shape of the answer: Jev returns a probability for every option and a confidence, so code can decide what to auto-accept; the LLM returns the label it picked.
Jev and the LLM agreed with the curators about 92% of the time, a statistical tie. When I read the records where Jev disagreed most confidently, nine of the first thirty were the curators' mistakes: a pancreatic cancer cell line filed under fat tissue, a kidney line filed under breast, a paediatric brain tumour filed under muscle. The confident disagreements are a cheap way to audit a database.
What made it cheaper was not the prompt. A lookup table of cell-line names answered 39% of records with no model call and no errors, and a second look at only the low-confidence rows fixed most of the rest. Together: accuracy up from 93% to 96% on the fresh sample, cost down by more than half.
| Wording variant (500 dev rows) | Correct | Words per record | $ per 1,000 |
|---|---|---|---|
| Full option descriptions | 470 | 1,578 | 0.066 |
| Short descriptions | 468 | 1,021 | 0.043 |
| Option names only | 424 | 600 | 0.025 |
| Short, 10 samples per request | 461 | 796 | 0.033 |
A few words per option is enough; long descriptions cost money and buy nothing. Bare names lose nine points because some options carry conventions a name cannot (a cell line files under its tissue of origin).
A research assistant that writes answers with citations can still misquote its sources. The task: for each cited sentence, read the source passage and decide whether it supports the sentence, is only partly supported, or is not supported.
My current fact-checker asks an LLM this question and takes about ten seconds per answer, which is too slow to run before the answer is shown to the reader. A checker that runs in under a second changes what the product can do.
Truth by construction: take answers that were checked clean, then plant one known error in a copy of each (a flipped negation, a changed number, a swapped citation, an overstatement). 35 planted errors and 147 clean sentences. A good checker catches the planted errors without flagging the clean ones, so both numbers are reported at a matched false-alarm budget. The no-model baseline is a regex that compares numbers.
* best on this metric. (tie): the accuracy difference is within noise on a paired test.
The one place the two disagree. Jev caught the flipped "does / did not" and said so on both questions; the LLM called it "partial" and gave the negation question 0.0. On the full set Jev caught 4 of 4 negations and the LLM judge 2 of 4.
The vendor's own documentation warns that Jev is weak on negation and numbers. On negation it caught 4 of 4 (the LLM caught 2 of 4). On numbers, a ten-line regex catches all 7 of 7 with one false alarm, so nobody should ask a model to compare numbers anyway. Used exactly as shipped, Jev flags twice as many clean sentences as the LLM; the fix is to set the threshold on its probability instead of taking its top answer, and that is a real integration cost someone has to pay.
A benchmark lesson on the way: the test set described its clean sentences as "verified clean". There was no verification step in the code. The controls were written by the same model that judges them.
A catalog of 3,000 regulatory-genomics tools needs each one tagged with the kinds of work it does: motif discovery, peak calling, 3D chromatin, and so on, up to 20 categories, often several per tool. From the tool's name, description and registry annotations, answer twenty yes/no questions, one per category.
The catalog already uses an LLM for this, with the usual JSON parsing and a documented habit of blank or invented category names. A model that structurally cannot invent a category is the obvious replacement, if it reads as well.
788 tools were tagged by hand; 150 of them tuned the wording and threshold, 638 were scored once. The baseline is the catalog's own keyword rulebook. Both models saw the same record text; Jev was also tried with the paper's abstract added.
* best on this metric. (tie): the accuracy difference is within noise on a paired test.
Same reading, same miss: both follow the registry's "Peak calling" tag and neither sees the peak-annotation role a curator knows about. Jev also hands back the near-misses (0.41) that a threshold or a reviewer can use.
Both models beat the rulebook clearly, and both plateau around F1 0.72. Ten rounds of tuning (wording, thresholds, adding the paper's abstract) moved Jev from 0.712 to 0.742 and no further. The categories both miss are the secondary ones: a tool that also annotates peaks as a side function. That is knowledge about the tool, not something written on the page, and a model that reads what is on the page cannot supply it.
Jev's yes/no probabilities never reach 0 or 1 (they span 0.02 to 0.96), and the best cut-off for "yes" was 0.85, not 0.5. Thresholds have to be tuned per task.
When a clinical lab finds a variant in a patient's DNA, it classifies it as pathogenic, uncertain or benign by working through a checklist of evidence types (functional studies, family data, population frequency, computational predictions) and combining them by a published rule. The task: from a written evidence summary, answer one yes/no question per checklist item; code then applies the standard combination rule.
This is the shape Jev's makers built their own benchmarks around: several records per case, code owns the decision rule, the model only answers narrow reading questions. If the approach works anywhere in biology, it should work here, and there is public truth at both levels.
1,000 variants classified by expert panels, 200 per class. The panel's own evidence text was used with its verdict and checklist codes stripped out (sentences naming another variant's classification were kept, since some criteria cannot be read without them). 150 variants tuned the wording, 850 were scored once. Baseline: a majority vote of what other labs had already said about each variant. Ceiling: the generic rule applied to the panel's own codes.
* best on this metric. (tie): the accuracy difference is within noise on a paired test.
A miss for both, and an instructive one: the summary describes reduced enzyme activity in patient cells (PS3), a second variant in trans (PM3) and a matching biochemical phenotype (PP4), but the panel's own phrasing did not use the words the questions were written with, and after the panel's verdict sentences were redacted neither model connected them. Wording tuned on 150 such texts moved Jev from 73% to 85% on this task; this record stayed a miss.
Both models beat the no-reading baseline by only a couple of points, and the reason is not the reading. Feed the panel's own checklist codes into the generic combination rule and it reproduces the panel's verdict only 87% of the time, because panels apply gene-specific exceptions. One variant was called pathogenic by its panel despite a "stand-alone benign" frequency; Jev read the frequency correctly and the rule overrode it. The fix is to put each gene's rules in code, which is what the expert panels publish.
| DeepSeek output format (150 dev variants) | Correct | Seconds each | $ per 1,000 |
|---|---|---|---|
| Probabilities, no thinking | 126 | 8.8 | 0.186 |
| Bare labels, no thinking | 129 | 4.4 | 0.150 |
| Probabilities, thinking on | 128 | 56.9 | 0.439 |
| Bare labels, thinking on | 127 | 27.6 | 0.311 |
"Thinking" mode bought nothing here at 6 to 13 times the latency. Bare true/false answers were the LLM's best format for this task.
Before tagging a tool, the catalog has to decide whether it belongs. The task: from a tool's record, decide admit or exclude. About 5,000 candidates: 3,069 admitted, and 430 looked at and kept out because they were really about protein structure, RNA modification, general read alignment, or were duplicates.
The kept-out tools are hard cases: they were only considered because they share the catalog's vocabulary. And the vendor recommends breaking a judgement like this into many small literal questions and letting code combine them. This task tests that advice directly.
Two ways of deciding, from the same request: one judgement question ("could this tool realistically be used in regulatory genomics?"), and 35 small ones ("is this about protein structure?", "is it a database rather than a tool?", one per category, and so on) combined by a hand-written rulebook and, separately, by weights learned from 300 tuning records. Duplicates were checked by code. The LLM was run in both of its output formats on a 1,000-record subset.
* best on this metric. (tie): the accuracy difference is within noise on a paired test.
The typical case behind the 85% vs 65 to 77% gap. The registry tag says "regulatory element prediction", the description says differential expression. Jev weighed the description; the LLM followed the tag and said "in scope" at 0.9.
Two results. First, the 35 small questions with learned weights scored exactly as well as the one judgement question (paired test: 159 records each way). My own hand-written rulebook over the same questions was much worse. Decomposition paid off only where code could decide outright: a name check caught 9 of 11 duplicate entries; the model caught 1.
Second, this was the one task where the LLM fell clearly behind. With bare yes/no answers it said "in scope" to almost everything (65% correct); with probabilities it ranked poorly (77%). Jev's probability on the single question ranked all 3,349 labelled tools well. The format that was the LLM's best on the variant task was its worst here, because most of these cases are borderline and true/false cannot say so.
ClinVar, the public database of clinical variant interpretations, shows a star rating per variant but not what the evidence is. The task: read each lab's written submission and record which kinds of evidence it reports (a functional experiment, family data, patients observed, population frequency, computational prediction), then summarise per variant.
A "pathogenic, one star" entry might rest on an experiment and twelve patients, or on a frequency lookup and a computer prediction. For a genome-explorer app I wanted that distinction on the variant page. This one is not a benchmark; it is the first thing built with Jev because the earlier tests said it would work.
Scope chosen to fit a small budget: 8,116 variants in 79 carrier-screening genes, restricted to the ones a genome is likely to actually carry, with up to two lab submissions each. 14,375 submissions, $0.41, zero failures. The six reads were validated first on 850 expert-panel texts where the panel's own checklist codes say which evidence types are present. The LLM ran the same validation and 1,000 of the lab submissions.
* best on this metric. (tie): within noise.
| Evidence type | Present in | Jev precision / recall | DeepSeek precision / recall |
|---|---|---|---|
| Functional study | 151 | 0.71 / 0.87 | 0.70 / 0.87 |
| Segregation in a family | 104 | 0.92 / 0.93 | 0.94 / 0.89 |
| De novo | 34 | 0.91 / 0.91 | 0.91 / 0.91 |
| Patients observed | 396 | 0.87 / 0.93 | 0.77 / 0.97 |
| Population frequency | 750 | 0.92 / 0.98 | 0.92 / 0.98 |
| Computational prediction | 559 | 0.94 / 0.93 | 0.92 / 0.93 |
Identical reading. Over 1,000 such submissions the two agreed on the evidence tier 92% of the time; the LLM's disagreements were mostly "patients observed" called on text that only implied it.
Half of these likely-to-be-carried pathogenic or conflicting variants rest on indirect evidence or none, in the text ClinVar holds. Among variants with conflicting classifications it is 61%. And of 418 variants whose submitters disagree, 236 disagreements are substantive and 182 are arguments over indirect evidence only, a distinction the star rating does not make.
Laya is an open-weight model (Apache licence) with the same three question types and almost the same request format as Jev. It runs on a laptop GPU with no API and no cost.
For private data, a hosted model is not an option at all; a local one with the same interface would be the only acceptable path. And a viral post claimed it was 50 times faster than Jev.
Same harness, same questions, same records as the other tabs. It finished the label-cleaning task (1,732 records) and part of fact-checking before the laptop's temperature stopped the run.
On the label-cleaning task it answered "Pluripotent stem cell" for 37% of records and was right half the time; on fact-checking it flagged three quarters of the clean sentences. Its documentation says the base checkpoints need fine-tuning for a task, and that matches. It also ran my laptop to 92 °C even single-threaded at a 20% duty cycle, so the other tasks stayed unrun.
The right use is as a fine-tuning target: let Jev label a few thousand records for a few cents, review the confident disagreements, and train the local model on the result. That is a separate project.
| Task | No-model baseline | Jev | DeepSeek V4 Flash, tuned | Verdict |
|---|---|---|---|---|
| Cleaning labels (accuracy) | 22.8% | 92.1% (95.9% tuned) | 91.5% | tie |
| Fact-checking (errors caught of 35) | 7 | 23 (26 tuned) | 25 | tie |
| Tagging tools (F1) | 0.646 | 0.722 (0.736 tuned) | 0.711 | tie |
| Genetic variants (accuracy) | 80.6% | 82.6% | 81.5% | tie |
| Catalog gate (accuracy) | — | 85.4% | 65.3% to 76.7% | Jev |
| Evidence table (precision, 6 types) | — | 0.71 to 0.94 | 0.70 to 0.94 | tie |
| Speed per item | 0.28 s, every task | 1.5 to 10 s | ||
| Cost, Jev as a share of the LLM's | 30% to 65% | 100% | ||
| Invalid answers | 0 in ~35,000 calls | 2 in ~7,000 |
Small, fast, constrained models are a good fit for one job: reading short text and making a bounded decision, at volume, with a probability you can act on. For anything that needs judgement across many records, put the judgement in one question and the bookkeeping in code.