Local LLM for Accountants: Running AI Offline

A local LLM runs a language model on your own computer. Download the model first, disable cloud features and test it offline before using client information.

Start with a small model for drafting or summarising. Choose larger hardware only when a harder task justifies it. Local processing changes where information goes, while the practice still needs to check the answer.

What a Local LLM is

A local large language model (LLM) generates answers using model files stored on your computer. The computer does the processing instead of sending each question to a hosted model.

A self-hosted LLM can also run on a server managed by your practice. That server might sit in your office or a cloud account. Self-hosted therefore describes responsibility for running it, without proving that processing happens in the office.

Prompts and answers can stay on the device when every part of the setup runs locally. A cloud model, online search or a connected document service changes that arrangement. Include chat history and backups when checking where information goes.

Model files contain the learned patterns used to generate text. Your prompt supplies the immediate task and any supporting information. Running a downloaded model does not automatically give it access to your folders or current tax guidance.

Hardware a Practice Needs

Memory determines which model can fit, while the processor and graphics hardware affect how quickly it answers. Leave room for your operating system and other work as well as the model.

System RAM is the computer’s working memory. VRAM is memory on a separate graphics card. Apple Silicon computers share a pool of unified memory between the processor and graphics hardware, so their memory figures need a different comparison.

Quantisation stores model numbers using fewer bits. It reduces memory use and can alter output quality. A four-bit model needs roughly half a byte per parameter for its weights, before additional memory and file overhead.

For example, nine billion parameters at four bits gives about 4.5 GB of raw weights. The actual download is larger. Running it needs extra memory for the conversation and processing, so download size is not a total memory requirement.

Illustrative computer	Sensible starting point	What to watch
Apple Silicon laptop with 16 GB unified memory	Quantised 4B model, then a 9B trial	Other applications and long conversations consume the same memory
Desktop with 32 GB RAM and 12 GB VRAM	Quantised 9B model	Confirm the complete workload fits the graphics card
Desktop with 64 GB RAM and 24 GB VRAM	Quantised 27B model with a modest context	Leave space beyond the weights for the conversation and runtime

These are pilot configurations, not guaranteed capacity figures. Context is the text a model holds for the current conversation. Increasing that text can make a previously workable model run out of memory.

Ollama’s published Qwen 3.5 downloads illustrate the difference. Its 4B version is about 3.3 to 4.0 GB, its 9B version 6.6 to 7.6 GB and its 27B version 17 to 20 GB. Formats differ.

For a 24 GB graphics card, start a 27B model with a short conversation. Try a smaller model if it spills into system RAM or responds too slowly. The best GPU is the supported card that meets your measured workload.

The Best Local LLM Models for Accounting Work

Start a practice pilot with Qwen 3.5 9B for short summaries and drafting. Try 4B where memory is tight and 27B where a harder document task warrants more memory. These are alternatives within one available model family.

The following developer-reported scores compare instruction following and long-context reasoning. IFEval tests compliance with instructions. LongBench v2 tests understanding and reasoning over long material. Neither score establishes Australian tax accuracy or performance on your quantised download.

Model	Parameters	IFEval score	LongBench v2 score	Pilot use
Qwen 3.5 4B	4 billion	89.8	50.0	Short drafts and summaries on limited hardware
Qwen 3.5 9B	9 billion	91.5	55.2	A starting comparison for practice correspondence
Qwen 3.5 27B	27 billion	95.0	60.6	Longer document tasks when memory permits

Qwen’s model cards report these results for its published evaluations. A higher score suggests a useful candidate for a pilot. It does not prove that the model will preserve every qualification in a client’s letter.

For summarising, compare the output with the original and mark omitted conditions. For drafting, use a standard practice template and check that the model adds no promises. Measure each task separately before choosing a larger model.

For reasoning with numbers, use a fictional reconciliation with a known answer. For example, $120,000 plus $12,000 plus $8,000 totals $140,000. Ask the model to identify the source amounts, then verify the calculation in your spreadsheet.

Treat fluent explanations of arithmetic as text to check. A language model can select the wrong amount or apply the wrong rule even when its final addition is correct.

Model rankings can change from month to month as developers release new weights and runtimes change. Quantisation and prompts also affect results. Reuse the same practice examples when comparing replacements, so a newer release has to demonstrate a useful improvement.

The open-source AI guide explains the licensing questions behind downloadable models.

How to Run a Local LLM

Run a local LLM by installing the runner, downloading one local model and checking that it answers with internet access disconnected. The procedure below follows Ollama 0.40.15 documentation for macOS and Windows.

Use a practice-managed computer and fictional information for setup. Ollama requires macOS 14 or newer on a Mac, or Windows 10 22H2 or newer. Install a supported graphics driver if you want GPU acceleration.

1. Install Ollama from its official download page. Open the desktop application and complete setup. In Terminal on macOS or PowerShell on Windows, run ollama --version to identify the installed release.
2. Disable Ollama cloud features. In your user folder, edit .ollama/server.json and set disable_ollama_cloud to true. Preserve any existing settings. Restart Ollama, then confirm its logs contain Ollama cloud disabled: true.
3. Download the small starting model by running ollama pull qwen3.5:4b. Wait for the download to finish. Then run ollama run qwen3.5:4b to open its local chat.
4. Keep the connection local. Retain the default address 127.0.0.1:11434. Avoid network sharing or remote connectors during this pilot. Run ollama ps while the model is active to inspect its CPU or GPU placement.
5. Prevent file syncing. Keep test documents and saved output outside cloud-synchronised folders. Disable any connected app’s chat sync and cloud fallback. Have your IT provider identify backup destinations before approving identifiable client information.
6. Disconnect Wi-Fi and any wired internet connection. Ask the model to summarise a short fictional paragraph. It must answer without a download or sign-in request. Reconnect only after that check finishes.
7. Compare the answer with the paragraph. Then run the known-answer reconciliation above. Record omissions and wrong amounts alongside the model tag and settings before approving a practice task.

The cloud setting belongs inside the existing JSON object. A new file containing only that setting looks like this.

{
  "disable_ollama_cloud": true
}

An offline answer proves the downloaded model can run without internet access. It does not establish what a separate app syncs when the connection returns. Review that app and your backup arrangement before moving beyond fictional information.

If the model fails to load, select a smaller download and shorten the conversation. If replies are slow, inspect CPU/GPU placement and memory use. If it requests a cloud connection, return to the local model selection and cloud settings.

Limits of Local Models

Local models trade hosted convenience for responsibility over hardware and maintenance. Your practice manages downloads and updates, while each computer limits the model size and conversation length available to staff.

A hosted service can run a larger model without buying a larger office computer. A local model can answer without an internet connection after setup. Compare the quality of the actual task before treating either arrangement as sufficient.

A downloaded model does not keep itself informed about changes to tax rules. Give it the relevant source text for a summary and open the original guidance before relying on a tax conclusion. Keep calculations in checked spreadsheets or dedicated calculation tools.

Set an owner for runtime updates and model changes. Keep the previous model details and repeat the same fictional examples after an update. A shared office server also needs permissions and enough capacity for simultaneous staff use.

Use the private AI guide when you need to compare local hardware with other deployment options. Keep the local pilot limited to tasks whose outputs your practice can check before anyone relies on them.