When LLMs fail at classification

I built Jeffy, an open-source text classifier that uses BGE-large embeddings + logistic regression. I wanted to know how it actually compares to LLMs, decision models, and other classifiers

So I benchmarked it against 15 models. 10 local LLMs (0.5B to 32B), the leading hosted decision model (Jev), two open-source decision models (Laya, GLiNER), and two traditional classifiers (SetFit, FastText)

tl;dr: on classification with many classes, a trained embedding + logistic regression wins. Not by a little. By a lot

The chart that tells the whole story

Here's what happens as you add more classes. At 2 classes (binary sentiment), everyone's competitive. By 4, Jeffy starts pulling ahead. At 77, everything else collapses

Speed

Speed matters when you're classifying in a loop. Emails arriving, chat messages streaming, API requests hitting. Here's every model on banking77 (the hardest task):

SetFit (6ms) and FastText (<1ms) are faster, but they trade too much accuracy for it. At 77 classes SetFit drops to 54% and FastText to 1%. Jev is the closest on both speed and accuracy (134ms, 80%), but it's a hosted API: your data leaves your machine and you pay per call. Jeffy runs entirely on your CPU

The setup

Three tasks, picked for class count:

• sst2 - binary sentiment (2 classes). The easy one
• ag_news - news topics (4 classes). Moderate
• banking77 - customer intent (77 classes). The real test

100 held-out samples per task. All LLMs via Ollama on the same machine, temperature 0, prompted with "reply with ONLY the category name." Decision models get the label set at call time. Jeffy, SetFit, and FastText trained on standard training splits

Full results by task

Every single test sample

Each square is one sample. Green = correct, red = wrong. Hover for the text and prediction. Watch the green disappear as class count goes up

Why decision models break down

Decision models like Jev and Laya process every option in a single forward pass. At 2 or 4 classes, that works great. At 77, the options don't fit the model's attention window cleanly. Labels overlap semantically ("card not working" vs "contactless not working" vs "card payment not recognised") and the model has to somehow pick between dozens of near-synonyms it's never seen training data for

A trained classifier doesn't have this problem. It saw labeled examples of each intent during training. It learned the embedding-space boundaries between them. The 77-class case isn't harder to run. Same BGE embedding, same logistic regression, just more output classes

Run it yourself

Every number here is reproducible:

# Install
pip install jeffy-classify datasets

# Pull an LLM to compare
ollama pull phi4

# Run the benchmark
python benchmarks/jeffybench.py --models phi4,mistral,llama3.1

# Single task
python benchmarks/jeffybench.py --tasks banking77 --models phi4

Bottom line

LLMs are great generalists. Decision models are elegant for simple choices. But when you have labeled data and more than a handful of classes, a trained classifier wins on accuracy, speed, cost, and privacy. It's not even close

Jeffy is open source, runs on CPU, and the whole model for 77 classes is under 5MB. No GPU, no API key, no data leaving your machine

Try 70 classifiers live · Source