Open weights
The trained numerical parameters are available to download, subject to a licence. This does not automatically disclose the training data, training code or full recipe.
What local AI actually is, which models are worth trying, what “open weights” really means, how much hardware you need and when running your own AI makes sense.
Most people encounter AI through a website or app: type a question, send it to a provider's servers, receive an answer. Local AI moves the model itself onto your computer. A program called an inference engine loads the downloaded model and generates responses on your hardware.
You can use it for drafts, summaries, classification, private notes, coding help and document analysis. You are responsible for the limitations, security and maintenance—and a local model does not automatically match the best paid cloud services.
Much of the confusion around downloadable AI comes from treating unrelated concepts as interchangeable.
The trained numerical parameters are available to download, subject to a licence. This does not automatically disclose the training data, training code or full recipe.
Software has an open-source licence when users have the relevant rights to inspect, modify and distribute its source code. AI models require separate scrutiny of weights, data and training procedure.
A 3B model has roughly three billion parameters. Parameter count is useful for estimating memory requirements, but bigger is not guaranteed to be better.
Stores weights at lower precision, such as 4-bit rather than 16-bit, shrinking memory and disk requirements—sometimes at a quality cost.
RAM is the computer’s main memory. VRAM is GPU memory. GPU execution is often faster, but CPU-only inference can work if you have sufficient system RAM.
Inference is using a finished model. Training or serious fine-tuning takes substantially more compute. Most beginners should download and run a pre-trained model first.
GGUF is a commonly used local-model file format supported by llama.cpp-based tools. A context window is how much text and other input the model can process in a session. MoE (mixture of experts) models use only part of their total network on each token, reducing compute but not necessarily the memory needed to keep model weights available.
A model may permit free downloads while withholding the exact list of books, websites, code repositories, licensed datasets and human-generated material used for training. A permissive model licence does not retroactively grant rights over the training dataset or guarantee every source was public domain.
Public web pages, books, software code, academic articles, multilingual text and carefully selected instructional examples can all be present. Some models also use synthetic outputs from other models, and multimodal systems can include images, video and audio.
Sources and proportions vary widely. Exact datasets may be partially documented, undisclosed or impossible to audit from the released weights alone.
Read the official model card, licence, technical report and dataset documentation. Look for listed sources, filtering rules, opt-out mechanisms, training cut-off, evaluation results and whether commercial use is explicitly permitted.
“Available on Hugging Face” is not a licence. “Apache 2.0” on the model repository is not proof that all training inputs were copyright-free.
The Common Pile project focuses on training data that is public domain or openly licensed, and associated Comma models demonstrate a more transparent approach. This makes them especially relevant to readers who want a documented training corpus. It does not mean all downstream model outputs are free of legal questions, nor that these models match leading commercial systems.
Check the current dataset and model cards before making legal or commercial claims. Copyright law differs by jurisdiction, including Australia.
These are representative families worth investigating—not a live leaderboard. A model family contains many sizes, variants and licence terms. Always select a specific released checkpoint and read its model card.
| Model family | Strengths / use case | Licence considerations | Beginner hardware fit |
|---|---|---|---|
| Qwen 3 / newer Qwen releases | General chat, multilingual work, coding and reasoning across multiple sizes | Varies by specific release; check model card | Small quantised variants can suit 8–16 GB systems |
| Gemma 3 / newer Gemma releases | General assistance, small-device options; some variants are multimodal | Google Gemma terms; not automatically an OSI open-source licence | Choose small variants first |
| Meta Llama 3.x / later Llama releases | General assistant tasks and broad ecosystem support | Custom Llama licence and use restrictions | Smaller variants for consumer computers |
| Mistral / Ministral / Small | General-purpose tasks, code and compact deployment options | Licences differ across checkpoints | Compact models only for entry-level hardware |
| DeepSeek-R1 distill models | Reasoning-oriented tasks; various distilled sizes | Inspect each derivative's base-model licence | Small distilled variants first |
| Microsoft Phi family | Compact models for constrained hardware and targeted tasks | Check individual release terms | Strong small-model category to explore |
| OpenAI gpt-oss | Open-weight reasoning models for capable local systems | Check official licence and hardware requirements for chosen release | Generally not the first choice for an 8 GB laptop |
| Comma / Common Pile | Training-provenance research and openly licensed data experimentation | Read both dataset and model licences | Depends on exact checkpoint |
Model families and availability evolve quickly. For exact specifications use Ollama’s model library or Hugging Face; check the original publisher's page too.
Good for learning the workflow and testing speed on an ordinary computer. Expect weaker reasoning, writing depth and factual reliability than large cloud models.
Often a useful balance of memory, speed and capability when you have about 16 GB RAM or an appropriately sized GPU.
Can improve quality but raises memory, power and latency requirements. Check whether the gain is worth the hardware.
Model size on disk is not the same as total runtime memory. The system also needs room for the operating system, inference engine, context cache and other applications. These are conservative planning tiers for 4-bit text models, not product guarantees.
| Your computer | Practical starting point | Expectations | Upgrade priority |
|---|---|---|---|
| 8 GB RAM, no dedicated GPU | About 1B–3B quantised | Usable for simple tasks; CPU speed may be limited | More RAM |
| 16 GB RAM, no dedicated GPU | About 3B–8B quantised | Reasonable beginner setup; speed depends heavily on processor | 32 GB RAM or GPU |
| 32 GB RAM, CPU or unified-memory system | About 7B–14B; some larger models may load | More flexibility; large CPU models can be slow | Accelerator / bandwidth |
| GPU with 8 GB VRAM | About 3B–8B quantised | Fast smaller-model inference if model + cache fit | 12–16 GB VRAM |
| GPU with 12–16 GB VRAM | About 7B–14B quantised; some larger options with trade-offs | Good midrange desktop option | 24 GB VRAM |
| GPU with 24 GB VRAM | About 14B–32B quantised, depending on architecture and context | Strong enthusiast category, still far from unlimited | More memory / multi-GPU |
A 32B 4-bit model can require roughly 18–24 GB for weights alone, depending on quantisation and file format. Long context windows increase cache use; multimodal inputs add overhead. Apple silicon unified memory is not directly equivalent to dedicated GPU VRAM.
Start with a small model and find out whether the responses and speed meet your needs. Don't buy a graphics card until you know which tasks you need and how often you'll run them.
Some Chromebooks can run Linux tools in Crostini, but ARM compatibility, available RAM, container storage and GPU access vary. A Chromebook running Linux is not equivalent to a dedicated AI workstation. Check architecture and build availability first.
Software can be free; hardware and electricity are not. These are illustrative planning ranges in Australian dollars, not verified current shop quotations.
| Path | Up-front AUD | Ongoing | Who should choose it? |
|---|---|---|---|
| Existing laptop or PC | $0 additional hardware | Electricity; usually low for intermittent use | Everyone starting out |
| RAM / SSD upgrade | ~$80–$350+ | Electricity | Upgradable desktop/laptop that is short on memory or space |
| Used GPU desktop | ~$700–$1,500+ | Electricity; repairs possible | Frequent users comfortable with hardware trade-offs |
| New midrange GPU workstation | ~$1,500–$3,000+ | Electricity | Regular users needing faster inference |
| High-VRAM enthusiast system | ~$3,000–$6,000+ | Higher power, cooling and maintenance costs | Large models and heavy daily usage |
| Cloud AI subscription | $0 additional hardware | Plan or API fees; may include limits | Users needing top models without equipment investment |
Power (kW) × hours × $/kWh = electricity cost
If a computer averages 150 watts while generating and runs 2 hours a day at an assumed $0.35/kWh, energy costs about $38.33 per year. That calculation excludes other computer use, idle energy, cooling, network charges, hardware depreciation and electricity-plan variations.
At 350 watts under the same assumptions, it is about $89.43 per year. These are examples, not measured figures for a specific computer.
Watch the hidden costs: storage for several model checkpoints, buying insufficient VRAM, cooling noise, maintenance time and the opportunity cost of troubleshooting. Compare your actual use with a cloud plan, not an imaginary unlimited cloud equivalent.
| Question | Local model | Proprietary hosted model |
|---|---|---|
| Privacy | Prompts can stay on-device if you disable network-dependent features | Prompts are sent to a service; privacy policies and enterprise controls vary |
| Offline availability | Yes, after setup and download, for supported tasks | Usually requires an internet connection |
| Quality on demanding tasks | Smaller models often lag; high-end local models can be capable | Leading cloud models often outperform typical consumer-local models |
| Speed | Depends on CPU/GPU and model size | Provider handles compute; network and service load still matter |
| Customisation | Strong control over models, prompts, tooling and local integrations | Usually easier onboarding; controls vary by service |
| Cost | Equipment and energy, typically no per-token fee for local inference | Subscriptions or metered API usage; no personal GPU purchase |
| Current information | Knowledge is frozen until updated or combined with retrieval/search | May provide integrated browsing or fresh tools, depending on product |
| Responsibility | You manage security, updates, reliability and legal compliance | Provider handles much infrastructure, but user responsibilities remain |
Private document workflows, offline access, repetitive classification jobs, coding experiments, model tinkering and predictable workloads are credible reasons to run locally.
Frontier-level reasoning, complex multimodal analysis, occasional demanding jobs and zero-maintenance setup can justify paying for a hosted service. A hybrid setup is often more rational than ideological loyalty to either side.
Two practical routes: a graphical desktop app or a command-line runner. Both can run downloaded models locally. Compatibility depends on your operating system and processor.
Install LM Studio, open its model browser, find a small model compatible with your computer, and download a quantised version. Load it and start chatting. Confirm the publisher, licence, RAM requirement and format before download.
Best for: beginners who prefer clicking over terminal commands.
Find your installed RAM, processor type (x86-64, ARM, Apple silicon) and dedicated GPU VRAM. Keep several gigabytes of free disk space for downloads.
Try a quantised 1B–3B model rather than a huge reasoning model. A model that responds in seconds is more useful for learning than a powerful model that stalls.
Use the same five prompts you use in paid AI: a rewrite, summary, coding question, reasoning problem and structured extraction. Compare speed, correctness and hallucinations.
If the model is too weak, test the next size. If it is too slow or swaps memory to disk, move down or use a smaller context window.
For document search and question-answering, investigate retrieval-augmented generation (RAG). RAG does not retrain the model and still requires verification, permissions and secure handling.
llama.cpp is a widely used local inference project for GGUF models. Open WebUI offers a browser-style interface and supports several back ends. These are useful after you understand the basics; additional layers also increase configuration and security complexity.
Benchmarks, model architecture, task fit and quantisation can matter more than headline size. Larger is not universally better.
A 5 GB download can require more than 5 GB RAM once loaded. Long conversations consume additional memory.
Local models can hallucinate citations, arithmetic, code behaviour and facts. Keep human review for consequential work.
Read the exact checkpoint licence before redistribution, commercial deployment or derivative work. Derivatives may inherit conditions.
Model download, extensions, online search, update checks or remote endpoints may still use the network. Audit sensitive workflows.
Try small models on existing hardware and quantify your actual workload before purchasing an expensive GPU.
Yes. CPU inference works with compatible software and adequate RAM. Apple silicon systems, AMD GPUs and other accelerators may also work, but performance and software support vary.
Not automatically. Some large open-weight models perform strongly on individual benchmarks, but premium proprietary systems frequently lead on difficult reasoning, reliable tool use, long-context and multimodal workflows. Test against your own tasks rather than relying on marketing comparisons.
Possibly. Linux support, architecture, storage and memory are the deciding factors. Small CPU-based models are more realistic than large GPU workloads on most Chromebooks.
No. Weight availability says little about rights in training sources. Seek explicit dataset provenance if that is a requirement.
Normally, running inference does not alter model weights. Some apps may save chat history or use persistent memory features. True fine-tuning is a separate process.
Often, but not always. It depends on the precise model, its licence, usage restrictions and jurisdiction. The model and training dataset may have different legal terms.
Downloading and installing typically require connectivity. After setup, supported inference tasks can be offline. Optional tools such as search and remote APIs need connectivity.
Start with a reputable, small, quantised instruct model listed by Ollama or LM Studio, then judge it on your real tasks. There is no single best model for every machine and workload.
Links favour official software, model publishers and primary documentation. Review update dates, model cards and licence terms at the destination. This article is a practical overview, not a regularly refreshed price comparison.