When LLMs fail at classification

Oct 2026 · Benchmark · Nico Brenner
Correction, 26 Oct 2026. The SetFit and FastText numbers in the original version of this post were misconfigured, in our favour. FastText had been run on library defaults (lr 0.1, 5 epochs) on a 77-class problem and scored 1.1%, below the 1.3% chance rate; tuned it gets 85.9%. SetFit had been run with an unidentified smaller model; with the documented base at 20 shots per class it gets 85.0%. The qwen3-32B row is withdrawn rather than corrected. Jeffy's own numbers are unchanged. The charts and text below now use the corrected figures, so the gap on banking77 is 8.4 points rather than 40, and the argument of the post is about LLMs and decision models rather than about trained classifiers. What went wrong and what changed

I built Jeffy, an open-source text classifier that uses BGE-large embeddings + logistic regression. I wanted to know how it actually compares to LLMs, decision models, and other classifiers

So I benchmarked it against 15 models. 10 local LLMs (0.5B to 32B), the leading hosted decision model (Jev), two open-source decision models (Laya, GLiNER), and two traditional classifiers (SetFit, FastText)

tl;dr: on classification with many classes, a trained embedding + logistic regression wins. Not by a little. By a lot

94.3% Jeffy on 77 classes
80% Best competitor (Jev)
118ms Local CPU, no API
$0 Per prediction

The chart that tells the whole story

Here's what happens as you add more classes. At 2 classes (binary sentiment), everyone's competitive. By 4, Jeffy starts pulling ahead. At 77, everything else collapses

The crossover happens between 2 and 4 classes. By 77, even Jev (the best hosted decision model) gets 80% vs Jeffy's 94%. GLiNER gets 56%, Laya 38%. No LLM breaks 70%, regardless of size

Speed

Speed matters when you're classifying in a loop. Emails arriving, chat messages streaming, API requests hitting. Here's every model on banking77 (the hardest task):

SetFit (46ms) and FastText (<1ms) are the real competition here, and at 77 classes they land at 85.0% and 85.9% against Jeffy's 94.3%. FastText is the fastest thing in the benchmark by two orders of magnitude, and it ships an 800 MB model to do one task. Jev is closest on both speed and accuracy (134ms, 80%), but it's a hosted API: your data leaves your machine and you pay per call. Jeffy runs entirely on your CPU

The setup

Three tasks, picked for class count:

100 held-out samples per task. All LLMs via Ollama on the same machine, temperature 0, prompted with "reply with ONLY the category name." Decision models get the label set at call time. Jeffy and FastText trained on standard training splits; SetFit on 20 examples per class, because its contrastive stage is quadratic in training-set size. Classifier numbers are benchmark v2, measured on the full evaluation splits

Full results by task

Every single test sample

Each square is one sample. Green = correct, red = wrong. Hover for the text and prediction. Watch the green disappear as class count goes up

Correct Wrong

Why decision models break down

Decision models like Jev and Laya process every option in a single forward pass. At 2 or 4 classes, that works great. At 77, the options don't fit the model's attention window cleanly. Labels overlap semantically ("card not working" vs "contactless not working" vs "card payment not recognised") and the model has to somehow pick between dozens of near-synonyms it's never seen training data for

A trained classifier doesn't have this problem. It saw labeled examples of each intent during training. It learned the embedding-space boundaries between them. The 77-class case isn't harder to run. Same BGE embedding, same logistic regression, just more output classes

Run it yourself

Every number here comes from JeffyBench, which has the full results by task, the per-sample grid, and the latency numbers. It is reproducible end to end:

# Install
pip install jeffy-classify datasets

# Pull an LLM to compare
ollama pull phi4

# Run the benchmark
python benchmarks/jeffybench.py --models phi4,mistral,llama3.1

# Single task
python benchmarks/jeffybench.py --tasks banking77 --models phi4

Bottom line

LLMs are great generalists. Decision models are elegant for simple choices. But when you have labeled data and more than a handful of classes, a trained classifier wins on accuracy, speed, cost, and privacy. It's not even close

Jeffy is open source, runs on CPU, and the whole model for 77 classes is under 5MB. No GPU, no API key, no data leaving your machine