Your LLM is probably overkill for text classification

I keep seeing the same pattern. Someone needs to classify text (route support tickets, tag documents, detect intent) so they prompt an LLM. It works in the demo. A few months later the complaints are about latency, cost, and malformed JSON breaking things downstream

There's a simpler approach that gets better results on classification: freeze a sentence embedding model and train a logistic regression head on top. That's it. No fine-tuning, no GPU, no prompt engineering

I'll show both approaches on the same task so you can see the difference

The task: banking77

Banking77 is a customer intent dataset with 77 classes. Things like "lost_or_stolen_card", "card_payment_not_recognised", "contactless_not_working". Many labels are semantically close, which is what makes it a real test

This is the kind of problem that shows up everywhere: support ticket routing, email classification, voice assistant intent detection. Lots of categories, subtle differences between them

The LLM approach

Here's how you'd classify text with an LLM. Using OpenAI as the example since that's what most people reach for:

# LLM classification with OpenAI
import openai
import json

client = openai.OpenAI()

labels = [
    "activate_my_card", "age_limit", "apple_pay_or_google_pay",
    "atm_support", "automatic_top_up", "balance_not_updated_after_bank_transfer",
    "balance_not_updated_after_cheque_or_cash_deposit", "beneficiary_not_allowed",
    "cancel_transfer", "card_about_to_expire", "card_acceptance",
    # ... 66 more labels ...
    "virtual_card_not_working", "visa_or_mastercard",
    "why_verify_identity", "wrong_amount_of_cash_received",
    "wrong_exchange_rate_for_cash_withdrawal"
]

def classify_with_llm(text):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[{
            "role": "user",
            "content": f"""Classify this customer message into exactly
one of these categories: {json.dumps(labels)}

Message: {text}

Reply with ONLY the category name, nothing else."""
        }]
    )
    return response.choices[0].message.content.strip()

result = classify_with_llm("I was charged twice for the same transaction")
# → "card_payment_not_recognised" (maybe, if the JSON doesn't break)

Problems with this:

• You're stuffing 77 labels into the prompt. That's a lot of tokens
• The model has to parse the meaning of each label every time
• Output format is unpredictable. Sometimes it adds quotes, sometimes it explains itself
• Each call costs money and takes 1-15 seconds
• Your data goes to an external API

The embeddings + LR approach

Here's the same task with Jeffy:

# Classification with frozen embeddings + logistic regression
pip install jeffy-classify

from jeffy import Engine

engine = Engine()
result = engine.predict("banking77", "I was charged twice for the same transaction")

print(result["label"])       # → "card_payment_not_recognised"
print(result["confidence"])  # → 0.72
print(result["latency_ms"])  # → 118

Three lines. No prompt, no API key, no output parsing. The embedding model (BGE-large) runs locally via ONNX, and the classifier is logistic regression coefficients trained on the standard banking77 training set. The whole thing runs on CPU

Side by side

client = openai.OpenAI()

labels = ["activate_my_card",
  "age_limit", ...]  # 77 labels

response = client.chat
  .completions.create(
    model="gpt-4o-mini",
    messages=[{
      "role": "user",
      "content": f"Classify: ..."
    }]
  )

label = response.choices[0]
  .message.content.strip()
# hope it's valid

from jeffy import Engine

engine = Engine()

result = engine.predict(
  "banking77",
  "I was charged twice"
)

label = result["label"]
score = result["confidence"]
# always structured JSON

Benchmarks on banking77 (77 classes)

I ran this benchmark on a MacBook Air M2 with 16GB RAM. All LLMs via Ollama at temperature 0. 100 held-out test samples

Why does the LLM struggle here?

LLMs are generalists. They weren't trained specifically on banking intent labels. When you give them 77 options like "card_not_working", "contactless_not_working", and "card_payment_not_recognised", they have to figure out the difference from the label names alone

A trained classifier saw labeled examples during training. It learned the embedding-space boundaries between "lost my card" and "card won't tap" and "charged twice." It's not guessing from label names. It knows

This matters more as class count grows. At 2 classes (positive/negative sentiment), LLMs do fine. By 77 classes, they collapse

But what about GPT-4o / Claude?

Hosted LLMs are better than local ones at this, sure. But the fundamental problem stays: you're parsing generated text, paying per token, sending data to an external API, and dealing with seconds of latency per call. A recent paper ("The Embedder's Dilemma", COLM 2026) benchmarked 10 LLMs against 26 embedding models across 37 tasks and found embedding models beat LLMs on classification specifically, with LLMs costing up to 1,431x more

When to use each

Train on your own data

If you don't have a pretrained classifier for your task, training one takes seconds:

# Train a custom classifier
from jeffy import Engine

engine = Engine()

texts = ["reset my password", "can't log in", "upgrade my plan", "cancel subscription", ...]
labels = ["account_access", "account_access", "billing", "billing", ...]

engine.train("my_support_task", texts, labels)

result = engine.predict("my_support_task", "I forgot my login")
# → {"label": "account_access", "confidence": 0.91}

The logistic regression head retrains in seconds on CPU. No GPU, no fine-tuning loop, no hyperparameter search. You can retrain it as new labeled data comes in

Try it

pip install jeffy-classify
jeffy-serve

Open localhost:8400 and you have 68 pretrained classifiers running locally. Or try them online at playground.jeffyclassify.com