Till not too long ago I hadn’t tried working fashions domestically for fairly some time, the frustration
had simply at all times been too excessive after I did it. A couple of month in the past although, I dove again in – there
have been simply too many claims on the market to disregard, about how far they’ve come, the way it’s now a lot
extra possible to run them, and the way a few of them have turn into actually good at coding. So that is
my private expertise with utilizing them, on and off, over the previous 4 weeks or so.
On this memo, I’ll begin with a extra common introduction and undergo the elements that affect the
viability of those fashions for coding. In a follow-up memo, I’ll describe my precise experiences in additional element.
Scope
My essential curiosity is how helpful they’re for coding, and never simply auto full, however
agentic coding. Secondly, I am involved in how prepared and usable they’re extra
broadly, for builders who do not wish to dive right into a bunch of specs and additional tooling and
tweaks to make it work.
By way of {hardware}, I’ve been working fashions on these two machines:
- Apple M3 Max, 48GB RAM
- Apple M5 Professional, 64GB RAM
Components that affect viability
There are a myriad of things at play that may affect the outcomes, which makes it fairly
a tedious enterprise to guage which setup works finest below the useful resource constraints we
have. It additionally makes it very exhausting to filter the alerts from the noise when individuals share their
successes with these fashions on-line.
I discovered it notably baffling that within the automated eval setup, one mannequin clearly delivered higher outcomes on the stronger machine (not simply velocity, however higher code!), regardless of all different settings being the identical.
I am going to begin with a abstract, earlier than going into particulars of every of those elements.
prefills
(system immediate, instruments, …)
various instrument schemas affect…
gives templates to enhance
higher {hardware} match results in higher…
differ in optimisation for…
Determine 1: The various elements that may
impression the outcomes – faucet or hover over bins to spotlight arrows.
- Runnability: RAM is the core constraint. I used fashions between 15-25GB, context
home windows of most 64K, harnesses OpenCode and Pi with zero Abilities and MCP servers energetic) - Response velocity: Is impacted by many elements, however was fairly good for some fashions, and leaps and bounds from the place it was a yr in the past. Setups: LM Studio + 4BIT quantization + M3 Max/M5 Professional + GGUF/MLX each
- Viability for agentic coding: Instrument calling was tough nonetheless, the fashions typically failed, however can often self-recover from their failures. This can be a key element of agentic coding particularly. With out it, you possibly can nonetheless go the old school means of copying and pasting from a chat window after all. And small fashions are undoubtedly much more viable for auto full than agentic use.
- High quality of outcomes: Depends upon the duty (extra on that within the subsequent memo), however clearly nowhere close to the kind of functionality we will get from the large fashions. Total it was very hit or miss. I solely checked out correctness of performance, did not go into code high quality.
RAM
The mannequin weights should match into the obtainable RAM, or extra particularly, VRAM. In the event that they
do not, the runtime both crashes (occurred to me as soon as!) or drops to unusably gradual
speeds.
On Apple Silicon, just about all RAM is accessible to the GPU with no separate VRAM
restrict,
this may be completely different in different machine configurations.
Impacts: Mannequin runnability; Velocity of responses
My expertise: On the machine with 48GB, I ran fashions between 8 and virtually 30GB.
The 30GB stretches it a number of course, particularly when the context window will get added,
the 15-25GB measurement is extra comfy and I haven’t got to shut fairly as many different
functions. On the machine with 64GB, I as soon as ran a mannequin that was 48GB massive – which
went nice at first, however then it shortly crashed…
Processing energy
Extra cores typically means sooner token era, however structure issues too, and
newer chip generations can shut the hole even with fewer cores. This can be a powerful one to
evaluate between machines with out diving a lot deeper into the main points of every
configuration
and structure.
Impacts: Velocity of responses
My expertise: On each the M3 Max and the M5 Professional, all fashions I ran had fairly an
spectacular velocity, in comparison with the place they have been a yr in the past. Velocity degrades although the longer a dialog will get. I’m okay utilizing them on the present velocity for some duties – if the standard of the output have been acceptable.
Reminiscence bandwidth
Reminiscence bandwidth is a bottleneck for token era, figuring out how briskly information strikes
between RAM and the compute items.
Impacts: Velocity of responses
My expertise: Each the M3 Max and the M5 Professional I used have a virtually an identical
bandwidth of ~300 GB/s, so I do not actually have a comparability to one thing else. However as
talked about earlier than, the velocity of all fashions I attempted was fairly acceptable.
Variety of parameters
The parameter rely principally represents the scale of a mannequin’s discovered information and
capabilities. Extra parameters typically imply higher output high quality, but additionally a bigger
file
measurement.
Impacts: Quantity of RAM wanted; High quality of outcomes
My expertise: With 48GB, I used fashions round 30B parameters, +/- 5B. The
greatest mannequin I loaded on the 64GB machine was Qwen3 Coder Subsequent 80B (MoE), which solved
the duty I gave it quite a bit higher than the smaller fashions – however then crashed after I
continued the dialog.
Reasoning capabilities
Reasoning fashions undergo a “chain of thought” course of earlier than responding, which helps
with advanced multi-step duties, however also can generate considerably extra tokens and gradual
responses down.
Impacts: Complexity of the duties; Velocity of response; Context window measurement (and
subsequently want for RAM)
My expertise: All the fashions I attempted had reasoning capabilities, they usually have been switched on by default all through most of my experiments. Nevertheless, I
typically observed them stepping into infinite circles within the reasoning chain, particularly
the smaller ones. (“Wait, …”, “Really, …”, “However wait, …”) So I additionally did a couple of runs of my automated setup with reasoning off – and lo and behold, it is not solely sooner (which was to be anticipated), but additionally carried out the identical to barely higher! An excellent reminder that reasoning is just not at all times essential, and might generally even be counterproductive.
Instrument calling capabilities
For agentic use, a mannequin should be capable of reliably emit structured instrument name syntax that
matches the schema the harness expects. Fashions that weren’t particularly skilled or
fine-tuned for instrument calling typically produce malformed calls.
Impacts: Capacity to make use of agentic harnesses
My expertise: This was a standard concern with the fashions I attempted, although they
might typically self-correct and get better from a failed instrument name (e.g. utilizing unsuitable parameter
names like file.path as a substitute of filePath).
Format
GGUF is the usual format for llama.cpp-based runtimes like LM Studio and Ollama, and
has by far the biggest mannequin library. MLX is Apple’s personal framework constructed particularly
for
Apple Silicon and might be sooner, however fewer MLX-formatted fashions can be found on the
second.
Impacts: Velocity of responses
My expertise: I attempted each codecs for one or two of the fashions, however I
personally did not really feel a lot of a distinction. That could possibly be as a result of
unstructured kind of analysis I did – then again, as any person expertise
researcher would inform us, the
perceived velocity is finally what issues, not what the clock says…
Quantization
Quantization compresses mannequin weights to cut back the file measurement, buying and selling some high quality for
a
a lot smaller RAM footprint. The extent of quantization is often marked in mannequin names
and
descriptions as This autumn / Q6 / Q8, or 4BIT / 6BIT / 8BIT, with decrease numbers that means increased
compression. The most recent buzz not too long ago have been QAT (Quantization-Conscious Coaching)
variants
of fashions, that are skilled with quantization simulated in the course of the coaching. They’re
purported to protect high quality higher than commonplace quantization.
Impacts: Quantity of RAM wanted; Velocity of responses; High quality of responses
My expertise: All the fashions I downloaded have been at This autumn / 4BIT, I did not get
round to making an attempt completely different variations but. I additionally have not gotten round to making an attempt a QAT
one.
Structure
MoE (Combination of Consultants) fashions have a big whole parameter rely however solely
activate a subsection of their weights at inference time, so a 35B MoE mannequin wants
considerably much less RAM and might run sooner than a 35B dense mannequin.
Impacts: Quantity of RAM wanted; Velocity of responses
My expertise: The Qwen3.6 35B MoE mannequin was by far giving me one of the best stability between variety of parameters and RAM utilization, and subsequently runnability and high quality of outcomes. This could possibly be as a result of MoE structure, I am undecided. The structure may also clarify my expertise of getting higher coding skills out of the mannequin on the 64GB machine than the 48GB – it may be loading extra consultants there? I am undecided if that is true, but it surely’s the one wise speculation I’ve to date.
Context window measurement setting
Context window measurement consumes RAM on high of mannequin weights by means of the KV cache, which
grows with context size. The default measurement configured within the runtimes is much too small
for agentic coding, it must be set to not less than 32K, if not 64K.
Impacts: Dimension and complexity of duties; Quantity of RAM wanted; Velocity of responses;
Capacity to make use of reasoning
My expertise: I attempted to see how little I might get away with. For small duties
I might generally work with 32K, however typically I needed to enhance to 64K, in order that appears to be
a superb default minimal. Because the fashions themselves have been already pushing the boundaries of
my obtainable RAM, I am undecided how how way more I might nonetheless enhance it, even on the
64GB machine… So whereas many of those fashions in concept help a bigger window,
truly utilizing it’s restricted by the reminiscence constraints.
Listing of fashions used
The thrill for coding has been throughout Qwen3 and Gemma 4, so these have been those I
went
for.
Qwen3
- Qwen3.6 35B-A3B MoE This autumn GGUF (22 GB)
- Qwen3.6 Coder Subsequent 80B MoE GGUF (45 GB)
Gemma 4
- Gemma 4 12B This autumn GGUF (7.5 GB)
- Gemma 4 26B 4BIT MLX (15.6 GB)
- Gemma 4 31B 4BIT MLX (29 GB)
Runtime
The runtime handles mannequin discovery, configuration, and loading. It additionally determines the
sensible query of how we combine our harness with the mannequin. Often this occurs by
beginning an online server that gives a spread of typical APIs that the harnesses help,
and the localhost URL of that internet server is then configured as a mannequin
supplier within the harness. Most generally supported by harnesses is the OpenAI API, however Claude
Code e.g. expects Anthropic’s
Claude API .
Impacts: Ease of configuration and discoverability; ease of integration with
harnesses; velocity of responses
Determine 2: The “Developer” view in LM Studio, displaying a
working server and lots of the elements described to date (supplier url, mannequin measurement, APIs,
context window measurement configuration)
My expertise: Whereas I’ve used different runtimes up to now, I’m at the moment again
utilizing LM Studio, primarily for its person expertise. There are many ins and outs of which runtime is probably the most optimised for which {hardware} and what kind of fashions, to get much more velocity out of it. However pondering again to the broader viability of working native fashions, person expertise performs an enormous position
for that. For what it is value, probably the most ceaselessly talked about various from my colleagues was oMLX.
Harness (Claude Code, OpenCode, Pi, …)
Coding harnesses can fluctuate considerably in how a lot overhead they inject into the context
window (system immediate, variety of instruments), which turns into extra of an issue domestically the place
we’re
so useful resource constrained. Our personal expanded harness round that additionally makes a distinction,
e.g.
what number of expertise or MCP servers are energetic. An outline of every of them can be despatched to
the
mannequin, and once more, take up area within the context window.
I discussed above that small fashions nonetheless battle with instrument calling – and it most likely
does not assist that every harness has barely completely different schemas for the essential instruments. Let’s
take enhancing a file for instance:
- Pi:
old_textandnew_text(see
right here) - OpenCode:
oldStringandnewString(see
right here) - Claude Code:
old_stringandnew_string(not less than that is what
it says after I requested it)
Lastly, not all harnesses simply help the combination of native fashions. Open supply
instruments are often the go-to, however Claude Code can be pointed at native suppliers. GitHub
Copilot appears to help it for his or her CLI, and I feel it is potential in Cursor as effectively
to override the OpenAI base URL and level it to localhost.
Impacts: Dimension of context window wanted; Instrument calling success; Integratability
My expertise: In my makes an attempt, I used OpenCode and Pi. I prevented Claude Code because it
apparently would burden the context
window fairly a bit.
Arising
In my subsequent memo, I’ll dive extra deeply into the kinds of duties I gave the fashions, and what I skilled.
A preview of my general conclusions: Utilizing small fashions like that is nonetheless fairly messy and exhausting to guage. It was a irritating expertise to come back to conclusions, because the outcomes rely upon so many issues. I’d subsequently say that it is nonetheless not prepared for a easy “plug and play” expertise for builders who do not wish to spend an excessive amount of time on it.
Nevertheless, primarily based on this expertise I do have a go-to mannequin that I am utilizing domestically now, which is Qwen3.6 35B MoE. It supplied one of the best stability of functionality, velocity and RAM footprint amongst what I attempted.
Extra to come back!
