LOCALAI / FIELD GUIDE
2026 edition · Australian guide

Run powerful AI.
On your own computer.

What local AI actually is, which models are worth trying, what “open weights” really means, how much hardware you need and when running your own AI makes sense.

Updated 17 November 202618-minute guidePrices in AUD where stated
No subscription required to experimentPrivacy-first optionsBeginner friendly

Start with a free model ↓

$0 model usage feesPossible with free runners and compatible weights; electricity and equipment still cost money
8–16 GB RAMEntry point for very small quantised text models; more memory enables larger models
Offline capableAfter installation and download, many tasks never have to leave your device
01 / The proposition

Your own AI, without sending every prompt to a server

Most people encounter AI through a website or app: type a question, send it to a provider's servers, receive an answer. Local AI moves the model itself onto your computer. A program called an inference engine loads the downloaded model and generates responses on your hardware.

You can use it for drafts, summaries, classification, private notes, coding help and document analysis. You are responsible for the limitations, security and maintenance—and a local model does not automatically match the best paid cloud services.

Laptop and desktop PC running a stylised neural network locally
02 / Decode the jargon

Seven terms you need to know

Much of the confusion around downloadable AI comes from treating unrelated concepts as interchangeable.

Ownership & access

Open weights

The trained numerical parameters are available to download, subject to a licence. This does not automatically disclose the training data, training code or full recipe.

Transparency

Open source

Software has an open-source licence when users have the relevant rights to inspect, modify and distribute its source code. AI models require separate scrutiny of weights, data and training procedure.

Model size

Parameters (B)

A 3B model has roughly three billion parameters. Parameter count is useful for estimating memory requirements, but bigger is not guaranteed to be better.

Efficiency

Quantisation

Stores weights at lower precision, such as 4-bit rather than 16-bit, shrinking memory and disk requirements—sometimes at a quality cost.

Hardware

RAM vs VRAM

RAM is the computer’s main memory. VRAM is GPU memory. GPU execution is often faster, but CPU-only inference can work if you have sufficient system RAM.

Usage

Inference vs training

Inference is using a finished model. Training or serious fine-tuning takes substantially more compute. Most beginners should download and run a pre-trained model first.

What do labels like GGUF, context window and MoE mean?

GGUF is a commonly used local-model file format supported by llama.cpp-based tools. A context window is how much text and other input the model can process in a session. MoE (mixture of experts) models use only part of their total network on each token, reducing compute but not necessarily the memory needed to keep model weights available.

03 / Copyright, training data & trust

“Open weights” is not “copyright-free”

A model may permit free downloads while withholding the exact list of books, websites, code repositories, licensed datasets and human-generated material used for training. A permissive model licence does not retroactively grant rights over the training dataset or guarantee every source was public domain.

What models can learn from

Typical training materials

Public web pages, books, software code, academic articles, multilingual text and carefully selected instructional examples can all be present. Some models also use synthetic outputs from other models, and multimodal systems can include images, video and audio.

Sources and proportions vary widely. Exact datasets may be partially documented, undisclosed or impossible to audit from the released weights alone.

Where to look

Evidence that matters

Read the official model card, licence, technical report and dataset documentation. Look for listed sources, filtering rules, opt-out mechanisms, training cut-off, evaluation results and whether commercial use is explicitly permitted.

“Available on Hugging Face” is not a licence. “Apache 2.0” on the model repository is not proof that all training inputs were copyright-free.

A more provenance-conscious path

The Common Pile and Comma models

The Common Pile project focuses on training data that is public domain or openly licensed, and associated Comma models demonstrate a more transparent approach. This makes them especially relevant to readers who want a documented training corpus. It does not mean all downstream model outputs are free of legal questions, nor that these models match leading commercial systems.

Check the current dataset and model cards before making legal or commercial claims. Copyright law differs by jurisdiction, including Australia.

Copyright and commercial-use warning: “Open”, “open-weight”, “free to download”, “open-source” and “trained on public-domain data” are five distinct claims. Verify each separately. This page is educational, not legal advice.
04 / Model landscape

Which model should you download?

These are representative families worth investigating—not a live leaderboard. A model family contains many sizes, variants and licence terms. Always select a specific released checkpoint and read its model card.

Model familyStrengths / use caseLicence considerationsBeginner hardware fit
Qwen 3 / newer Qwen releasesGeneral chat, multilingual work, coding and reasoning across multiple sizesVaries by specific release; check model cardSmall quantised variants can suit 8–16 GB systems
Gemma 3 / newer Gemma releasesGeneral assistance, small-device options; some variants are multimodalGoogle Gemma terms; not automatically an OSI open-source licenceChoose small variants first
Meta Llama 3.x / later Llama releasesGeneral assistant tasks and broad ecosystem supportCustom Llama licence and use restrictionsSmaller variants for consumer computers
Mistral / Ministral / SmallGeneral-purpose tasks, code and compact deployment optionsLicences differ across checkpointsCompact models only for entry-level hardware
DeepSeek-R1 distill modelsReasoning-oriented tasks; various distilled sizesInspect each derivative's base-model licenceSmall distilled variants first
Microsoft Phi familyCompact models for constrained hardware and targeted tasksCheck individual release termsStrong small-model category to explore
OpenAI gpt-ossOpen-weight reasoning models for capable local systemsCheck official licence and hardware requirements for chosen releaseGenerally not the first choice for an 8 GB laptop
Comma / Common PileTraining-provenance research and openly licensed data experimentationRead both dataset and model licencesDepends on exact checkpoint

Model families and availability evolve quickly. For exact specifications use Ollama’s model library or Hugging Face; check the original publisher's page too.

First experiment

Try a 1B–3B model

Good for learning the workflow and testing speed on an ordinary computer. Expect weaker reasoning, writing depth and factual reliability than large cloud models.

Everyday local work

Try 7B–9B quantised

Often a useful balance of memory, speed and capability when you have about 16 GB RAM or an appropriately sized GPU.

Heavier work

Try 14B–32B and beyond

Can improve quality but raises memory, power and latency requirements. Check whether the gain is worth the hardware.

05 / Hardware

How much computer do you actually need?

Model size on disk is not the same as total runtime memory. The system also needs room for the operating system, inference engine, context cache and other applications. These are conservative planning tiers for 4-bit text models, not product guarantees.

Your computerPractical starting pointExpectationsUpgrade priority
8 GB RAM, no dedicated GPUAbout 1B–3B quantisedUsable for simple tasks; CPU speed may be limitedMore RAM
16 GB RAM, no dedicated GPUAbout 3B–8B quantisedReasonable beginner setup; speed depends heavily on processor32 GB RAM or GPU
32 GB RAM, CPU or unified-memory systemAbout 7B–14B; some larger models may loadMore flexibility; large CPU models can be slowAccelerator / bandwidth
GPU with 8 GB VRAMAbout 3B–8B quantisedFast smaller-model inference if model + cache fit12–16 GB VRAM
GPU with 12–16 GB VRAMAbout 7B–14B quantised; some larger options with trade-offsGood midrange desktop option24 GB VRAM
GPU with 24 GB VRAMAbout 14B–32B quantised, depending on architecture and contextStrong enthusiast category, still far from unlimitedMore memory / multi-GPU

A 32B 4-bit model can require roughly 18–24 GB for weights alone, depending on quantisation and file format. Long context windows increase cache use; multimodal inputs add overhead. Apple silicon unified memory is not directly equivalent to dedicated GPU VRAM.

The cheapest approach

Use the machine you own

Start with a small model and find out whether the responses and speed meet your needs. Don't buy a graphics card until you know which tasks you need and how often you'll run them.

Chromebook caveat

Linux container limitations

Some Chromebooks can run Linux tools in Crostini, but ARM compatibility, available RAM, container storage and GPU access vary. A Chromebook running Linux is not equivalent to a dedicated AI workstation. Check architecture and build availability first.

06 / Prices in Australia

What does local AI cost?

Software can be free; hardware and electricity are not. These are illustrative planning ranges in Australian dollars, not verified current shop quotations.

PathUp-front AUDOngoingWho should choose it?
Existing laptop or PC$0 additional hardwareElectricity; usually low for intermittent useEveryone starting out
RAM / SSD upgrade~$80–$350+ElectricityUpgradable desktop/laptop that is short on memory or space
Used GPU desktop~$700–$1,500+Electricity; repairs possibleFrequent users comfortable with hardware trade-offs
New midrange GPU workstation~$1,500–$3,000+ElectricityRegular users needing faster inference
High-VRAM enthusiast system~$3,000–$6,000+Higher power, cooling and maintenance costsLarge models and heavy daily usage
Cloud AI subscription$0 additional hardwarePlan or API fees; may include limitsUsers needing top models without equipment investment
A simple electricity example

Estimate running costs yourself

Power (kW) × hours × $/kWh = electricity cost

If a computer averages 150 watts while generating and runs 2 hours a day at an assumed $0.35/kWh, energy costs about $38.33 per year. That calculation excludes other computer use, idle energy, cooling, network charges, hardware depreciation and electricity-plan variations.

At 350 watts under the same assumptions, it is about $89.43 per year. These are examples, not measured figures for a specific computer.

Watch the hidden costs: storage for several model checkpoints, buying insufficient VRAM, cooling noise, maintenance time and the opportunity cost of troubleshooting. Compare your actual use with a cloud plan, not an imaginary unlimited cloud equivalent.

07 / Trade-offs

Local AI versus proprietary cloud AI

QuestionLocal modelProprietary hosted model
PrivacyPrompts can stay on-device if you disable network-dependent featuresPrompts are sent to a service; privacy policies and enterprise controls vary
Offline availabilityYes, after setup and download, for supported tasksUsually requires an internet connection
Quality on demanding tasksSmaller models often lag; high-end local models can be capableLeading cloud models often outperform typical consumer-local models
SpeedDepends on CPU/GPU and model sizeProvider handles compute; network and service load still matter
CustomisationStrong control over models, prompts, tooling and local integrationsUsually easier onboarding; controls vary by service
CostEquipment and energy, typically no per-token fee for local inferenceSubscriptions or metered API usage; no personal GPU purchase
Current informationKnowledge is frozen until updated or combined with retrieval/searchMay provide integrated browsing or fresh tools, depending on product
ResponsibilityYou manage security, updates, reliability and legal complianceProvider handles much infrastructure, but user responsibilities remain
Beware the privacy shortcut: installing a local AI front-end does not guarantee private processing. Cloud-provider options, telemetry, plug-ins, remote embeddings, web search and shared endpoints can still transmit data. Check the actual configuration.
Strong reasons to stay local

Control and repeat use

Private document workflows, offline access, repetitive classification jobs, coding experiments, model tinkering and predictable workloads are credible reasons to run locally.

Strong reasons to use cloud

Capability and convenience

Frontier-level reasoning, complex multimodal analysis, occasional demanding jobs and zero-maintenance setup can justify paying for a hosted service. A hybrid setup is often more rational than ideological loyalty to either side.

08 / Get started today

The easiest way to run your first model

Two practical routes: a graphical desktop app or a command-line runner. Both can run downloaded models locally. Compatibility depends on your operating system and processor.

Option A · Visual interface

LM Studio

Install LM Studio, open its model browser, find a small model compatible with your computer, and download a quantised version. Load it and start chatting. Confirm the publisher, licence, RAM requirement and format before download.

Best for: beginners who prefer clicking over terminal commands.

Option B · Simple command line

Ollama

Install Ollama and choose a small current model from its library. Copy the model’s exact run command from its official listing.

ollama run MODEL_NAME

Replace MODEL_NAME with a verified model tag from the live library.

1

Check your hardware

Find your installed RAM, processor type (x86-64, ARM, Apple silicon) and dedicated GPU VRAM. Keep several gigabytes of free disk space for downloads.

2

Start smaller than you think

Try a quantised 1B–3B model rather than a huge reasoning model. A model that responds in seconds is more useful for learning than a powerful model that stalls.

3

Test real tasks

Use the same five prompts you use in paid AI: a rewrite, summary, coding question, reasoning problem and structured extraction. Compare speed, correctness and hallucinations.

4

Move up one size at a time

If the model is too weak, test the next size. If it is too slow or swaps memory to disk, move down or use a smaller context window.

5

Add your own documents carefully

For document search and question-answering, investigate retrieval-augmented generation (RAG). RAG does not retrain the model and still requires verification, permissions and secure handling.

More control: llama.cpp and Open WebUI

llama.cpp is a widely used local inference project for GGUF models. Open WebUI offers a browser-style interface and supports several back ends. These are useful after you understand the basics; additional layers also increase configuration and security complexity.

09 / What goes wrong

Common mistakes that waste money and time

Buying for parameter count

Benchmarks, model architecture, task fit and quantisation can matter more than headline size. Larger is not universally better.

Confusing storage with memory

A 5 GB download can require more than 5 GB RAM once loaded. Long conversations consume additional memory.

Trusting polished answers

Local models can hallucinate citations, arithmetic, code behaviour and facts. Keep human review for consequential work.

Ignoring licences

Read the exact checkpoint licence before redistribution, commercial deployment or derivative work. Derivatives may inherit conditions.

Assuming fully offline

Model download, extensions, online search, update checks or remote endpoints may still use the network. Audit sensitive workflows.

Overspending too early

Try small models on existing hardware and quantify your actual workload before purchasing an expensive GPU.

10 / Frequently asked questions

Short answers to common questions

Can I run AI without an NVIDIA GPU?

Yes. CPU inference works with compatible software and adequate RAM. Apple silicon systems, AMD GPUs and other accelerators may also work, but performance and software support vary.

Is local AI as good as ChatGPT, Claude or Gemini?

Not automatically. Some large open-weight models perform strongly on individual benchmarks, but premium proprietary systems frequently lead on difficult reasoning, reliable tool use, long-context and multimodal workflows. Test against your own tasks rather than relying on marketing comparisons.

Can I run AI on a Chromebook?

Possibly. Linux support, architecture, storage and memory are the deciding factors. Small CPU-based models are more realistic than large GPU workloads on most Chromebooks.

Are open-weight models copyright-free?

No. Weight availability says little about rights in training sources. Seek explicit dataset provenance if that is a requirement.

Will an AI model learn from my conversations?

Normally, running inference does not alter model weights. Some apps may save chat history or use persistent memory features. True fine-tuning is a separate process.

Can I use a model commercially?

Often, but not always. It depends on the precise model, its licence, usage restrictions and jurisdiction. The model and training dataset may have different legal terms.

Does using a local AI require the internet?

Downloading and installing typically require connectivity. After setup, supported inference tasks can be offline. Optional tools such as search and remote APIs need connectivity.

What is the best model for a beginner?

Start with a reputable, small, quantised instruct model listed by Ollama or LM Studio, then judge it on your real tasks. There is no single best model for every machine and workload.