I built Jeffy, an open-source text classifier that uses BGE-large embeddings + logistic regression. I wanted to know how it actually compares to LLMs, decision models, and other classifiers
So I benchmarked it against 15 models. 10 local LLMs (0.5B to 32B), the leading hosted decision model (Jev), two open-source decision models (Laya, GLiNER), and two traditional classifiers (SetFit, FastText)
tl;dr: on classification with many classes, a trained embedding + logistic regression wins. Not by a little. By a lot
Here's what happens as you add more classes. At 2 classes (binary sentiment), everyone's competitive. By 4, Jeffy starts pulling ahead. At 77, everything else collapses
Speed matters when you're classifying in a loop. Emails arriving, chat messages streaming, API requests hitting. Here's every model on banking77 (the hardest task):
SetFit (46ms) and FastText (<1ms) are the real competition here, and at 77 classes they land at 85.0% and 85.9% against Jeffy's 94.3%. FastText is the fastest thing in the benchmark by two orders of magnitude, and it ships an 800 MB model to do one task. Jev is closest on both speed and accuracy (134ms, 80%), but it's a hosted API: your data leaves your machine and you pay per call. Jeffy runs entirely on your CPU
Three tasks, picked for class count:
100 held-out samples per task. All LLMs via Ollama on the same machine, temperature 0, prompted with "reply with ONLY the category name." Decision models get the label set at call time. Jeffy and FastText trained on standard training splits; SetFit on 20 examples per class, because its contrastive stage is quadratic in training-set size. Classifier numbers are benchmark v2, measured on the full evaluation splits
Each square is one sample. Green = correct, red = wrong. Hover for the text and prediction. Watch the green disappear as class count goes up
Decision models like Jev and Laya process every option in a single forward pass. At 2 or 4 classes, that works great. At 77, the options don't fit the model's attention window cleanly. Labels overlap semantically ("card not working" vs "contactless not working" vs "card payment not recognised") and the model has to somehow pick between dozens of near-synonyms it's never seen training data for
A trained classifier doesn't have this problem. It saw labeled examples of each intent during training. It learned the embedding-space boundaries between them. The 77-class case isn't harder to run. Same BGE embedding, same logistic regression, just more output classes
Every number here comes from JeffyBench, which has the full results by task, the per-sample grid, and the latency numbers. It is reproducible end to end:
# Install
pip install jeffy-classify datasets
# Pull an LLM to compare
ollama pull phi4
# Run the benchmark
python benchmarks/jeffybench.py --models phi4,mistral,llama3.1
# Single task
python benchmarks/jeffybench.py --tasks banking77 --models phi4
LLMs are great generalists. Decision models are elegant for simple choices. But when you have labeled data and more than a handful of classes, a trained classifier wins on accuracy, speed, cost, and privacy. It's not even close
Jeffy is open source, runs on CPU, and the whole model for 77 classes is under 5MB. No GPU, no API key, no data leaving your machine