A coding assistant that was never born

Soup promises to train an 8B language model on a 4 GB graphics card from a single configuration file. My PC has 6 GB of video memory and 32 GB of RAM, so the numbers seemed to add up. The plan was to build my own coding assistant, a model that would help me program from my own machine, and the real goal was to learn the technique.

Exams in a box with no network

Before training anything you need to know where you start from, and that calls for exams. We set up exams that run the code of every answer inside a container with no network, like marking papers in a room with no windows. Three untrained models got their baseline score on two public benchmarks and on an exam of my own with ten questions.

Phase 0 baseline

Model	HumanEval+	MBPP+	My own exam
Qwen2.5-Coder 1.5B	44.5%	66.1%	4/10
Qwen2.5-Coder 3B	65.9%	71.4%	Not measured
Qwen2.5-Coder 7B	70.7%	79.1%	9/10

The 3B performs almost like the 7B at less than half the size, and that was the first lesson, to measure before assuming that bigger is better.

Then came the training data. Of 200,000 examples about 62,000 good ones remained, and we removed 62 that looked too much like the exams, because a student who has seen the questions proves nothing. That is where the second lesson turned up. Our “normal” and “high quality” datasets also differed in the kind of problems they held, and an experiment is only worth something if it compares things that differ in a single variable. We caught it before training.

The memory the card does not have

Soup’s headline feature, layer streaming, is the one that promises to fit an 8B model on a 4 GB card, and it works. On my PC it gave exactly the same result as the normal method. What is fragile is everything around it. We had to work around several bugs in Soup and in the pieces it relies on, and one of them copied 1 GB to the card for every generated word.

And real memory is smaller than nominal memory. Windows keeps part of the card, some parts of the model do not compress, and the Linux that runs inside Windows caches files in RAM that count as free inside it and as taken for Windows. That last one stopped the tests twice.

A specialist with no trade

With the technique learned, the question that should have come first was still there. A small local model does not compete with the big tools as a generalist, and it only makes sense as a specialist in a narrow task. So we changed course towards droneobservatory, the code behind EODI, and measured its real running cost.

It did not pay off there either. The cost is about 39 USD a month, and 80% of it comes from the expert model writing news records, which is the hardest task and needs the text of the articles, which is not stored. Replacing the whole base model would save about 5 USD a month at most. What would save the most there is not training anything, but reviewing the rules that send records to the expert, the “critical” and “low confidence” ones, and that improvement is still pending.

The technique worked. What never showed up was a deliverable that justified going on, and without knowing what you want to deliver and what for, experiments end up as loose tests and there is no way to know when to stop. So I stopped.