How to save on AI API calls — a local news classifier that runs on the processor

The surest saving on AI APIs does not come from a cheaper model but from the calls you no longer make.

Labelling news does not need a model that writes, but one that recognises. We moved this work from a generative cloud model to a small, local, verifiable system, and the real cost was not computing power but data discipline: who is allowed to teach it, and how you know when it is wrong.

This analysis describes the architecture in broad strokes, what we tried and what failed, what worked, and the lessons we paid for along the way. We do not publish our internal performance figures: they depend on our own news flow and on a short measurement window, and out of context they would say more than we know.

The problem: a small task done with a big tool

olivLaw news items get event labels from a closed list of seven categories: motion of censure, coalition change, protest, military action, call for elections, judicial event, and the release of a macroeconomic indicator. Other parts of the system rely on them: a story's importance, its market impact, the simulations, and the “ⓘ” card in the feed.

Until now, every news item was sent to a large language model, which read the text and answered with the labels in JSON. The solution is convenient because it needs no training data, but it has three weaknesses. It depends on external providers, and when one fails or throttles requests, news items pile up in a queue. It pays for text generation on a task that needs no generation. And every answer has to be checked, because a generative model can return anything.

What failed: a local generative model on the processor

The first attempt to bring the task in-house was the obvious one: the same kind of model, but an open, mid-sized one run locally. On a server without a graphics card, the result was poor. Each answer took so long that it exceeded the time limit, and the model kept the processor so busy that it slowed down other heavy work on the same machine, such as the speech synthesis for our podcasts.

The lesson was simple: memory buys a larger model, not speed. On a processor, a mid-sized generative model produces a few tokens per second, however much memory the machine has. For a label from a closed list, generating text, even short text, is the most expensive route.

The architecture: two local sources and an arbiter

The new system has three parts, all on the same server.

1. A small encoder, distilled from a large model. Encoders are models that do not write: they turn text into a numerical representation and classify it directly. We started from mmBERT-small, a multilingual encoder published in September 2025 by researchers at Johns Hopkins under the MIT licence, trained on more than 3 trillion tokens in more than 1,800 languages, Romanian included. It has 140 million parameters, nearly a hundred times fewer than the local generative model we had tried. We trained it on the labels the large model had already set, then exported it to a portable format (ONNX), at full precision, so that it runs on the processor on a capped number of threads.

2. A neighbour memory. Every olivLaw news item already has a vector, a numerical “fingerprint” of its meaning, computed by an embedding model that runs locally. We put the news items whose label is trusted into a separate table, together with their fingerprints, and indexed the table for fast similarity search. When a new item arrives, we look up the closest items already labelled and let them vote. Closer neighbours weigh more, and those too far away do not vote at all.

3. A deterministic arbiter. Each of the two sources gives a probability for every label. The arbiter combines them with weights fixed in advance, applies a threshold per label, and flags as uncertain the cases that sit close to a threshold. If one source is missing (say the model failed to load), the arbiter works with the other and says so. It is all ordinary code, with no language model, and easy to test.

The result is written separately from the official label. For now, the local classifier runs in shadow: it sees the same news, sets its own labels, and we continuously compare the two. Moving it into production is an editorial decision, not an automatic one.

Why a memory, when there is already a trained model

We measured both sources separately, on the same news items. On its own, the neighbour memory is weaker than a trained model. It is not a better classifier but something else, with three qualities the model lacks.

First, it adapts without retraining. A new subject, say a conflict that starts today, enters the memory with the first reliably labelled items, and the next items find it immediately. The model would learn it only at its next training run.

Second, it shows its evidence. For each label we keep the closest items that voted. An editor can see why a story was marked “protest”: it resembles these five, which were protests.

Third, it takes human corrections on the spot. A label corrected by a person enters the memory with the highest priority and can no longer be overwritten by any model.

The rule that keeps the system honest

A system that learns by itself has a well-known trap: if it feeds its own predictions back into its memory, it ends up confirming its own mistakes. A wrong label attracts similar items, which get the same wrong label, which in turn enter the memory. The error reinforces itself.

That is why only two kinds of labels enter the memory: those set by a large model we have declared trusted, by provider and version, and those set by a person. The local classifier's predictions never do. The trusted list lives in a reviewed configuration file, not in code, and if the memory stops receiving new labels, a monitoring signal says so.

The recipe: how to save on API calls

What we learned applies to any team that pays per call for repetitive classification, extraction or labelling. The steps, in order of how much they bring:

1. Separate recognition from generation. Ask, for every call: is the answer a choice from a closed list, or new text? Choices (category, sentiment, language, “yes or no”) are a classifier's job, not a writing model's. Only new text truly needs a generative model.

2. Use the large model as a teacher, once. Let it label a few thousand real examples, keep with every label the model that set it, then train a small encoder on them. The paid calls become an investment that ends, not a monthly rent.

3. Reuse what you already compute. If you already have embeddings for search, you already have half a classifier. A simple layer on top of them, or a lookup of the nearest labelled neighbours, costs almost nothing.

4. Send only the uncertain cases to the API. A local model that knows its own uncertainty can decide the clear cases by itself and pass on only those close to a threshold. That way you pay only for the hard questions, not for all of them.

5. What can wait, send as a batch. The large providers offer asynchronous batch APIs: the answer does not come at once, but the price is much lower. At Anthropic, for example, a batch costs half the price of a regular call, and most batches finish in under an hour. Overnight summaries or relabelling an archive do not need an answer in seconds.

6. Do not pay twice for the same question. The same story picked up by ten sites, the same request retried after an error: a content-keyed cache and an idempotency key remove repeated calls.

7. Measure before you cut. Every call should leave a trace with the task, the model, the duration and the cost. Without it you do not know what is worth moving in-house, and you do not notice when a cheap provider becomes expensive through retries.

Lessons learned

A consistent teacher beats more data. The older labels came from several models, because the fallback chain switched models whenever one failed. A model trained on fewer labels, all from the same “teacher”, did at least as well as one trained on several times more mixed labels. Every model has its own habits about where it draws the line between categories. Mix the habits, and the student learns the noise. Since then, we keep with every label the model that set it.

What you already have is often enough. The first temptation was a new vector database as a separate service. But the fingerprints of all our news items were already in the main database, which knows how to search them. A new service would have brought a second copy of the data, synchronisation, backups and one more point of failure. We used what was there.

Start with the simplest option. Before training the model, we put a plain linear layer on top of the fingerprints we already had. It trains in under a minute. The fine-tuned model, which took hours of computing, at first came out only level with it, not above. A simple baseline tells you what the extra effort is actually worth and is often enough as a first step.

Rare categories are the real problem. Protests and military actions appear every day, and the model recognises them well. Motions of censure or calls for elections are rare, and training examples are few. Here the local model errs more often than the large one and will most likely need explicit, human-written rules over a closed vocabulary. That is not a failure of the method: it is where editorial judgement is worth the most.

Agreement with the teacher is not truth. Everything we have measured so far is how often the local classifier says the same thing as the large model. But the large model makes mistakes too. To know how correct the labels are, we need a set labelled by people, independent of both. We are building it, and until then we report results as agreement, not accuracy.

Measure on your own machine, and check every optimisation. Compressing weights to 8 bits brings large gains on processors with dedicated instructions and modest gains on older ones, like ours. For us it did worse: the compressed model came out slightly faster but started giving different answers from the original on practically every news item. We caught it only because we compared the two versions' decisions before launch. It now runs at full precision, a little slower but identical to the trained model. The speed figures in papers and on model pages are measured on other hardware: we redid them on our server, with the thread count capped, while the rest of the workload was running.

Model weights are untrusted data. The base model was published only in a format that can execute code when loaded. We loaded it once in a mode that accepts only arrays of numbers, converted it to a format with no executable code, and from then on worked only with that, inside an isolated container.

Shadow first, decision later. The local classifier writes separately, and five monitoring signals track whether it runs, whether it falls behind, how often it disagrees with the trusted label, whether the memory is still learning and whether the model loaded. A change that would replace the official label does not exist in the code yet. We will write it only once the measured agreement justifies it.

The causal chain

Cause: a recognition task (a label from a closed list) was treated as a generation task, by a large remote model. Mechanism: a large model labels enough examples once, a small model learns from them, and a memory of trusted examples covers new subjects and takes people's corrections. Effect: labels can be set locally, at any hour, without depending on an external provider, while the large model is kept for the uncertain cases. Counter-hypothesis: the quality gap on rare categories might stay too wide, in which case the local system remains a safety net for outages rather than a replacement for the official label. We will judge that on the human-labelled set.

Verifiable predictions

Prediction	Horizon	Probability	How it is checked	Verification source
The lab that published mmBERT (Johns Hopkins, CLSP) releases a new version or a successor of its multilingual encoder family	by 30.06.2027	40–55%	A new model in the lab's organisation or a new paper on arXiv	huggingface.co/jhu-clsp, arxiv.org
An open multilingual encoder with at most 500 million parameters reports better results than mmBERT-base on XNLI	by 30.06.2027	45–60%	The results table in its release paper	arxiv.org, huggingface.co
The mmBERT-small base model remains publicly available under the MIT licence	by 31.12.2027	85–95%	The model page and its stated licence	huggingface.co/jhu-clsp/mmBERT-small

Method and limits

Method. We trained the model and chose the thresholds on days other than those we tested on, so that the result describes news the model had not seen. We compared the same news items for every variant: the news fingerprints with a simple linear layer, the neighbour memory alone, the trained model alone, and the combination of the two. The probabilities in the predictions table are editorial estimates, in bands of at least ten percentage points.

Limits. The measurement window is short, a few days, and the rare categories have few examples in it. We measure agreement with the large model, not correctness. The results depend on our mix of sources and languages (Romanian, English and Russian). The diagnosis would be refuted if, on the human-labelled set, the local classifier erred systematically more often than the large model on the frequent categories too, not only on the rare ones.