How to Make a Text Classifier Using an LLM
Software often needs a bounded decision: detecting fraudulent transactions, checking whether a representative followed policy, or deciding whether a case should go to a person.
Our Classification-with-Confidence project works through that problem from end to end. It begins with a basic LLM text classifier, extracts the probability mass behind its possible labels, tests whether those raw scores match observed accuracy, calibrates them, and uses the result to decide which cases software can handle and which cases need review. It then fine-tunes the model for the specific task and measures the new confidence distribution.
The practical stake is bigger than getting a model to say Yes or No. A production system needs evidence about when that answer deserves action.
The Freud Detector
An early experiment gave us the basic shape. A text-generating model could be constrained to make a binary choice, and its first-token log probabilities could be adapted into a score for that choice. The adapter was awkward: labels had to appear among the provider's returned tokens, capitalization and whitespace mattered, and every model or API change could alter the result.
The prompt supplied the reveal:
Given the following quote, determine if it was said by Sigmund Freud.
Answer with either 'Yes' or 'No'.
Quote: {quote}
Was this quote by Sigmund Freud?
The Freud detector notebook was a small historical test of the technique. It also exposed the core limitation: a token probability looks precise before we know whether it corresponds to observed accuracy. That question became the focus of Classification-with-Confidence.
Count the whole class, not one spelling
The main project uses Meta's Llama 3.1-8B-Instruct model for binary sentiment classification. The prompt asks whether a piece of text is positive and requests yes or no as the answer. Because the model predicts tokens rather than abstract classes, one class can surface in several forms: Yes, yes, YES, y, and other configured equivalents.
Reading only the most likely token throws away the rest of that evidence. The project's classifier implementation instead adds the probabilities for every configured token that represents the positive class and does the same for the negative class:
positive score = Σ P(token) for every positive-label token
negative score = Σ P(token) for every negative-label token
prediction = class with the larger total
This matters when the model divides its probability across equivalent answers. In the repository's “I love this movie!” example, Yes and yes carry nearly all the mass between them. In its deliberately ambiguous “Best worst thing ever” example, the combined yes variants and no variants land close together. Aggregation recovers that difference much better than looking at the single winning spelling.
It still produces a raw model score. Probability assigned to unrelated tokens remains outside the two class totals, and a high total doesn't establish how often the classification is right.
Compare the score with outcomes
Classification-with-Confidence includes a labeled 10,000-example sentiment dataset organized into strong, medium, weak, and neutral positive and negative categories. The range was intentional: the project needed obvious examples and ambiguous examples to exercise different parts of the score distribution.
The repository reports a run of Llama 3.1-8B across 1,000 randomly selected examples. For each example, it recorded the aggregated class score and whether the predicted sentiment matched the label. It then grouped predictions into five-point score bands and compared each band's raw probability with its observed accuracy.
In the project's 1,000-example run, raw total probability generally sorted stronger from weaker predictions, but it overstated observed accuracy in important score bands. Bubble labels show the number of examples in each band.
The largest band contained 532 predictions with raw totals between 95% and 100%. Their observed accuracy in this run was 83.5%. That gap is the useful finding. The score carried information about difficulty, but its numeric value couldn't yet be read as a success rate.
These are project results from a constructed sentiment dataset and one model configuration. They don't establish calibration for another dataset, domain, prompt, model revision, or provider.
Calibrate before treating a score as confidence
Calibration learns a mapping from a raw score to observed correctness. Classification-with-Confidence implements Platt scaling and isotonic regression:
- Platt scaling fits a logistic curve, which is compact and smooth.
- Isotonic regression learns a monotonic mapping without assuming a sigmoid shape, which gives it more flexibility and a greater appetite for calibration data.
The repository's charts report that both mappings moved its reliability points closer to the diagonal where predicted confidence equals observed accuracy. Read that as a result of this experiment. It isn't a permanent property of Llama 3.1-8B, and it doesn't make a threshold a guarantee.
For a production evaluation, the calibration fit and the final measurement need separate data. Keep an untouched test partition, preserve meaningful slices, and rerun the check when the model, prompt, label definition, or input population changes. Otherwise the calibrator can describe the examples it already saw while failing on the next batch.
Route different confidence bands differently
Once the score has been tested and calibrated on representative data, it becomes an operating input. High-confidence, low-consequence cases might pass automatically. A middle band might go to a person. High-consequence decisions might always require review, regardless of score.
The threshold is a business decision because accuracy and coverage move together. Raising it sends fewer cases through automation and usually improves the observed accuracy of the remaining cases. Lowering it covers more volume and accepts more errors.
The project's raw-score run illustrates the coverage trade-off. The chart is diagnostic evidence from that sample, rather than a production service level: its 95% raw threshold selected 532 examples whose observed accuracy was 83.5%.
That is why the operating metric should be cost per accepted decision at a defined error tolerance. Inference, human review, retries, and the cost of mistakes all belong in the calculation. One global threshold is rarely enough; different questions and consequences deserve different policies.
Fine-tune the task, then measure again
The project next fine-tunes Llama 3.1-8B with LoRA on task-specific examples from the same sentiment dataset. Its current training code creates an 80/20 split before formatting the training data, and its comparison code prefers the saved held-out set when comparing the base and fine-tuned models.
The dataset intentionally contains a learnable domain pattern: sports contexts occur more often among positive examples, while workplace contexts occur more often among negative examples. That makes the experiment a demonstration of task-specific alignment. It also makes the boundary clear. The fine-tuned classifier is learning this project's labels and distribution; it isn't becoming a universally better sentiment model.
The repository reports higher task accuracy, lower calibration error, and more predictions in its high-confidence bands after fine-tuning. Its confidence-distribution chart shows the visible change:
In this project run, the fine-tuned model placed more predictions near the top of its score range. That can expand the useful high-confidence band only when held-out accuracy and calibration improve with it.
A pileup near 1.0 isn't automatically good news. A model can become more certain without becoming more correct. The fine-tuned path therefore has to repeat the same sequence: record raw probabilities, compare them with held-out outcomes, recalibrate, and choose thresholds from the resulting evidence.
The earlier classifier-native baseline
Before these LLM experiments, our semantic text classification project used Word2Vec, BERT, OpenAI Ada 2, and AWS Titan embeddings as inputs to logistic regression. The BERT path was BERT embeddings plus logistic regression, rather than BertForSequenceClassification. That conventional stack returned a task-specific class probability directly and helped frame the question explored here: how much machinery does it take to recover the same operating contract from a text generator?
What carries into production
Classification-with-Confidence provides a concrete progression:
- Constrain generation to a bounded classification answer.
- Aggregate the probability mass of equivalent label tokens.
- Compare raw scores with correctness on labeled examples.
- Fit and evaluate a calibration mapping.
- Route cases according to calibrated thresholds and consequence.
- Fine-tune for the task, then repeat the evaluation.
Every step depends on the data that supports it. A random sample from a deliberately balanced experiment is useful for learning the mechanism. Production evidence has to represent the traffic, edge cases, languages, clients, and failure costs the system will actually encounter. Calibration also drifts when those inputs change.
The technique depends on access to token probabilities. Local transformer models expose them; hosted providers vary by model and can change their APIs. Label tokenization and output behavior can change too. Version the whole path—model, prompt, token mapping, dataset, calibrator, and threshold—and re-evaluate it as a system.
That is the durable lesson. An LLM can be made to behave like a classifier, but useful confidence comes from measured outcomes and an operating policy around them.
Addendum — September 18, 2026
Since this article was first published, TypeSafe has introduced Jev, a model with a classifier-shaped interface for typed decisions and probabilities. Our companion article, Making Decisions Instead of Generating Text, examines that claim through the production lens developed here. The Freud notebook doesn't predict how Jev will perform in 2026. It explains why a native, probability-aware decision contract is worth testing: we spent years constructing one around models designed to generate text.