Local AI Is Not Free: Open-Weight Models, Hardware Cost and Accuracy

Open weights do not make inference free. They move the bill.

The API bill becomes hardware, electricity, maintenance, engineering time, and the opportunity cost of running a weaker or slower model. Privacy and control can justify that bill. A fantasy about “free local AI” cannot.

My decision model has four parts:

model quality -> memory fit -> useful speed -> total ownership cost

If one part fails, the deployment fails. A model that barely loads but runs too slowly is not a local solution. A fast model that cannot complete the work is not cheap. A private model that needs constant operational attention is not low maintenance.

This guide covers the major current open-weight decision set for one desk-side machine as of 7 August 2026. It is not every checkpoint on Hugging Face. That catalogue changes daily and would hide the decision. I include models that are current, materially different, supported by a primary model card, and plausible on 16GB to 128GB of local memory. I separately show frontier weights that are open but not honestly single-machine models.

“Content accuracy” is not a settled benchmark. Good prose has voice, structure, factuality, instruction-following, and judgement. I use MMLU-Pro or GPQA Diamond as a factual-reasoning proxy, not as a writing-quality score. For software work, I prefer SWE-bench Verified and Pro because they require repository changes. LiveCodeBench is listed only when a model vendor does not report SWE-bench.

Memory estimates are planning ranges for the model weights plus basic runtime overhead. They assume a roughly four-bit format where available. Long contexts need additional KV-cache memory. MoE models store all experts even though only a few are active per token. Active parameters reduce compute; they do not make the other weights disappear.

I use current UK prices, then label any complete-system estimate as an estimate. NVIDIA lists the RTX 5060 Ti 16GB at £349 MSRP and the RTX 5090 32GB at £1,799 MSRP. Those are card prices, not computers. Apple lists a complete 64GB MacBook Pro at £2,999. Dell lists a complete 128GB GB10 system at £6,125.27. Comparing a bare GPU with a finished Mac would be dishonest.

Open-weight and local are not synonyms.

I compare local and hosted inference over the same period.

monthly local cost
= purchase price / useful life in months
+ annual electricity / 12
+ annual maintenance reserve / 12
+ operator time

A £3,200 workstation over 36 months, with £600 a year in electricity and reserve, costs about £139 per month before my time. A £4,200 machine with £500 a year in operating cost is about £158 per month.

That creates a clean break-even test:

break-even accepted tasks
= monthly local ownership cost
/ hosted cost per accepted task

If a hosted model costs £2 per accepted task, the £139 local system needs about 70 accepted tasks each month just to match the direct spend. If the local model creates more review or repair work, the break-even point moves away. If data cannot leave the machine, the privacy requirement may settle the decision before cost does.

The wrong comparison is local hardware versus API tokens. The correct comparison is local accepted outcomes versus hosted accepted outcomes.

I would not select one model for both workloads from one leaderboard.

For content, I would test:

• factual precision against a source packet;
• unsupported claims and invented citations;
• instruction adherence;
• structural coherence over 1,500–3,000 words;
• editing distance from my accepted version;
• tokens per accepted article.

For software engineering, I would test:

• first-pass task completion;
• tests passed without weakening them;
• regressions introduced;
• tool and schema failures;
• reviewer minutes;
• wall-clock time;
• tokens or joules per accepted change.

The same model can win one and lose the other. Gemma 4's multimodal and content strengths do not prove repository repair ability. Qwen3-Coder-Next's SWE result does not prove editorial voice. Specialisation is not a defect. Pretending it does not exist is.

1. Run gpt-oss-20B or Gemma 4 12B QAT on hardware I already own. Measure quality and speed for two weeks.
2. Rent before buying. Run Qwen3.6-27B, Qwen3.6-35B-A3B, and Devstral Small 2 on rented instances with the exact local serving stack.
3. Define the quality floor. Use accepted content and accepted code changes, not vibes.
4. Choose the smallest memory tier that keeps 20% headroom. Include the real context window and concurrency.
5. Calculate 36-month ownership cost. Add electricity, maintenance reserve, and my operating time.
6. Buy only when utilisation or privacy makes the result obvious. If the spreadsheet needs heroic assumptions, the API is still cheaper.

My default is a 32GB system for one-person experimentation and Qwen3.6-27B as the balanced starting model. I choose Qwen3.6-35B-A3B when generation speed matters more than the last few benchmark points. I move to 64GB only when measured context, concurrency, or workspace requirements justify the extra memory. I treat 96–128GB as a niche capacity tier, not the automatic next step.

I do not buy local hardware to avoid a token bill. I buy it to gain control over a workload I have already measured.