Local Model Performance on an M5 Max

A few days ago, I read mamonas.dev's laptop benchmark, testing local models on a five year old, 2021 M1 Pro with 16GB of RAM. I ran a similar test, but with a 2026 M5 Max with 128GB of RAM.

TLDR

I ran 6 of the best local models possible on this hardware and compared them to frontier models in accuracy and speed. They actually did surprisingly well.

Why Run This?

I'm pretty interested in how local models perform these days, but since I happened to have just gotten a brand new MacBook M5 Max with 128 GB, I wanted to see how much better the latest semi-normal consumer hardware can do the same type of benchmark.

I say "semi-normal" because I'm not buying some crazy NVIDIA monster, just a higher end consumer Apple laptop. I mean, $5000 is a lot, but it certainly falls within reasonable for a senior software developer (or in my case, a toys obsessed CEO).

Mostly I want to see what capabilities will probably be widely available to most local laptops within a few years. If companies or developers have concerns about token costs or sending information to third parties, I wanted to see how close running local models on relatively commodity hardware comes to what we're currently enjoying, frontier-wise.

The Models

Now, in comparison to the momonas.dev post, the models that a 16GB M1 can run are pretty different than the ones available to an M5 Max with 128GB of unified memory.

In the original blog post, the models he was able to test were pretty small - two gemma models from Google (4:12b-it-qat, 4:e4b-it-qat) and three qwen models from Alibaba (3.5:4b, 3.5:9b, 2.5:7b) - all between ~3-7GB in size.

For my M5, we can load a larger class of newer models, so I tried everything I thought would be interesting. Specifically:

The Deepseek model was run via the antirez/ds4 project server, the rest were downloaded and run via Ollama. The pi coding agent harness was used to run the same suite of tests, which were derived from the mamonas.dev description.

Claude devised 22 problems in 6 categories: 5 DuckDB SQL, 4 agentic pytest bugfixes, 4 JSON extractions, 4 predict-the-output, 3 quick facts, 2 long-doc lookups. You can check out the benchmarking code here.

In addition to the local models, for comparison, I ran the exact same suite with Pi using two frontier cloud models - Opus 5 and GPT-5.6 Sol.

The Bottom Line

So, how did the local models do? Honestly, pretty good.

While the frontier models were both faster and more accurate, it's not by as much as I honestly expected. Here is the same speed vs accuracy chart that mamonas.dev used (top left is best).

It's interesting that the frontier models were not only the most accurate (to be expected) but also very fast, even though we're adding the networking overhead of the cloud round trips.

From the local model standpoint, both the new Muse Glimmer from Meta and the OpenAI GPT OSS had a good cross section of speed and accuracy. What is pretty cool is that the Muse model can run on as little as 32GB of RAM (gpt-oss needs a 96 or 128GB setup, more likely), so you don't even need my MacBook Beast to run it.

Another note is the Qwen3 Coder model is super fast because it's not a thinking model, but even so did a pretty good job on these example tests. Where it did fail seemed to generally be because it wasn't thinking (the prediction tests).

Accuracy

Most of the local models were very accurate for the test suite we came up with.

In contrast to the Mamona test where some of the models were failing half the tests, nearly all of the larger local models came very close to what the frontier models could do.

The interesting thing from the frontier models is where they failed - while Sol legit messed up some SQL tests and one of the prediction tests, the only other failures from both models were service related.

Sol had an overloaded service on one call and Opus refused to respond on a bunch of them because of "restrictions on violative cyber content". Probably because I was doing the exact same semi-sketchy looking prompt multiple times.

However, other than blocking the cyber crimes I was trying to commit, Opus 5 is the only model that actually made no mistakes on any of the test runs.

A Full Breakdown

So, just because I can, here is every model on every test — all 1,056 runs, by speed and accuracy:

First, some fun general observations.

You can see pretty clearly how fast the Qwen3 non-thinking model is and how generally good it is at several of the categories outside of the prediction runs.

Every local model is perfect on the 7,900-token runbook lookups, including Qwen3, which was on average faster than even the frontier models. This test was giving the model this 5,000 word runbook and asking stuff like "which port does the staging bastion listen on?".

The other fun thing is how very good Qwen3 was at writing SQL fast. It crushed every one of those problems and did so way faster than even the frontier models.

From a more general standpoint, most of the local models were also very good at the bugfix problems, which provided a small python file and test file and instructions like this:

"The test suite in this directory fails. Find and fix the bug(s) in cart.py so all tests pass. Do not modify test_cart.py. Run the tests with: ./run_tests.sh"

I would say that Muse was generally much faster than the other models at this type of task, though all of them solved them most the time (some looping timeouts). However, the frontier models were massively faster at the same problem and never made any mistakes.

Would you stop googling things?

One of the fun questions that the mamonas.dev post asked was "would you stop googling things?".

The speed question I actually cared about: for a quick factual lookup, does asking a local model beat a search engine?

He found that the gemma4:e4b and qwen2.5 models returned answers to questions like “what port does Postgres use” in under 1 second, while thinking models took up to half a minute.

None of the local models were anywhere near that slow. His qwen3.5:4b model averaged 38s for that question. In contrast, almost all of my local models beat Sol for quick questions like that - nearly everything averaged under 6 seconds.

Again, the Qwen3-Coder non-thinking model was lightning fast. Even the thinking models were on par with frontier stuff for quick googly-type answers.

Where the Models Failed

When the local models failed, what did they struggle with?

Python 2/3 Prediction test

The prediction test that fooled every model other than Opus (including Sol for half the runs) is this:

funcs = [lambda: i for i in range(3)]
print([f() for f in funcs])
i = 99
print(funcs[0]())

Muse got it right 1/3 of the time, GPT-OSS once, GLM-4.7 once. Opus is the only model that got it right all 6 runs. Deepseek and both Qwen models never got it right.

Apparently, the wrong answer was what Python 2 would print, not Python 3 due to a variable rebinding difference.

Hallucinating SQL dialects

Some of the models (gpt-oss and GLM) tried Postgres's TO_CHAR in a DuckDB query. Qwen3.5 invented a format_date() there.

Chmod

One of the questions was what the chmod rwxr-xr-- mode was. The correct answer is 754, but Muse specifically was quite bad at this for some reason. It, Qwen3.5 and GLM gave me 750, 744, 752...

The Costs

So, this is a bit of a difficult topic to really bottom line. When we're dealing with local models, obviously one of the semi-compelling things is that you don't need to pay for inference. But trying to calculate cost savings is a little tricky. Are we comparing it against subscriptions or advertised token list prices? How does caching affect this? It's not super straightforward.

However, the facts are that the 792 runs that I did for this benchmark (6 models x 22 tests x 6 runs each = 792 runs) moved approximately 6M input and 1M output tokens. So if we very literally apply that against API list prices for various frontier cloud services, the costs would be roughly:

In reality, the costs are almost certainly less for the big guys. Even through Anthropic API usage via Pi, the Opus run of it's 132 tests (~287k input / ~20k output) only cost $1.80, which would be about $10 for the whole run (not $55). Partially that's caching, partially that's better token efficiency. It's a difficult apples-to-apples here.

However, it's interesting to compare this to the economics of the original blog post. With my laptop and the newer models, I'm waiting less time and getting much better accuracy, entirely offline.

If it's the difference between even $10-20 in cloud costs for a night of work versus maybe $0.10 in electricity costs for my MacBook, if the results are even close (which at least in these simple tests, they clearly are), that becomes somewhat compelling.

What I’d Actually Keep

So, does this change anything? Actually, it already has.

I'm going to do some experiments with building whole applications with Muse, just to see how it does (perhaps another post later). However, I have already started using Ollama+Qwen3-Coder for some simple stuff, because it's so fast and apparently quite accurate.

The other interesting thing that I would like to dig into is security work. I was thrown off by the (clearly overreaching) cyber security violations of the Opus calls. It's nice to know I can ask these local models anything and they won't slap my hand for it.