Local LLMs, decision models, few-shot classifiers, and bag-of-words models. On structured classification, a trained embedding + logistic regression beats them all. The more classes, the bigger the gap.
On binary sentiment (2 classes), Jev leads at 96%, with LLMs and GLiNER close behind. By 4 classes Jeffy pulls ahead. At 77, everything else collapses.
On banking77, the hardest task. Jeffy runs on CPU; LLMs via Ollama; Jev via hosted API; others via Python APIs.
16 models: Jeffy, 10 LLMs (0.5B–32B via Ollama), 3 decision models (Jev, Laya, GLiNER2.5), SetFit, and FastText. 3 tasks, 100 samples each. LLMs prompted zero-shot at temperature 0. Jev via hosted API. Decision models given the label set at call time. SetFit and FastText trained on the same data as Jeffy.
Each square is one test sample. Green = correct, red = wrong. Hover for the input text, expected label, and prediction. Watch how the green squares disappear as class count grows — for LLMs and other classifiers.
13 tasks spanning 2 to 151 classes, all on held-out test sets.
JeffyBench is a reproducible benchmark. Clone the repo, install, and verify every number.
# Install Jeffy pip install jeffy-classify datasets # Pull the LLM you want to test ollama pull phi4 # Run the full benchmark python benchmarks/jeffybench.py --models phi4,mistral,llama3.1 # Or test on a single task python benchmarks/jeffybench.py --tasks banking77 --models phi4 # Results are printed as tables and saved to JSON