AI Hallucinations in Tax Work: Causes and Checks

An AI hallucination is an answer that presents invented or incorrect information as fact. In tax work, trace citations and check figures against their original sources before relying on the answer.

A ruling number can look convincing even when the ruling does not exist. A real ruling can also appear beside a conclusion it never supports.

What an AI Hallucination is

An AI hallucination occurs when generated content is false or unsupported, despite sounding certain. The term also covers an answer that contradicts the documents supplied to the model.

The NIST generative AI risk profile calls this confabulation. It describes confident false content and departures from the model’s inputs. A polished explanation does not establish that its claims have been checked.

A language model generates text using patterns learned during training and the context available when it answers. A likely sequence of words can resemble a tax explanation without accurately stating the law.

The model can combine a genuine legal principle with an invented citation. It can also accept an incorrect premise in your question and write an explanation around it.

Access to a search engine or document library helps the model find material. Errors remain possible when it retrieves the wrong source or draws a conclusion that the source does not support.

An old rate is not necessarily fabricated information. It becomes an incorrect answer when the model presents it as the rate for a different income year. Your review must catch both invented material and misapplied real information.

AI Hallucination Examples in Tax and Accounting

AI hallucinations in tax work can invent authority or apply real information to the wrong facts. These fictional examples show the difference and the review that exposes it.

Fictional Errors and Their Checks

Error	What the draft says	What the reviewer does
Invented ATO ruling	A supposed public ruling allows a deduction for every cost discussed in a client meeting. The model supplies a plausible ruling number.	Search the ATO Legal Database independently. Locate the document and read the passage said to support the deduction.
Superseded rate	A Division 7A schedule uses a prior income year’s benchmark interest rate as the current rate.	Match the ATO rate to each income year in the schedule and recalculate the affected amounts.
Wrong due date	A client email applies a general lodgement deadline to a client with a different agent-program due date.	Establish the client’s actual lodgement and payment obligations from ATO records. Check eligibility for any concession separately.
Made-up court case	A draft cites a supposed court decision allowing a deduction for private expenditure. Neither the case nor its quoted reasons can be found.	Search the court’s published judgments and a recognised legal database. Remove unsupported authority and reconsider the conclusion.

The ruling and case in these examples are fictional. No real ruling number or case citation is being presented as fake.

A source that exists can still be wrong for the client. A decision about a company does not automatically settle an individual’s deduction. Read the facts and reasoning before using the conclusion.

For a hypothetical loan review, suppose the spreadsheet balances but uses an earlier year’s rate. Arithmetic checks pass because the formula works. Matching the rate to the income year exposes the error that recalculating the same formula misses.

How Often AI Gets It Wrong

An AI hallucination rate describes performance on a particular test. It is not a fixed property of every answer produced by a model or product.

A study must decide what counts as an error and what it counts. Some assessments score whole answers. Others score individual factual claims or whether a citation actually supports a statement.

Refusals change the interpretation too. A system that declines difficult questions can produce fewer false answers while leaving more work unanswered. Compare error rates alongside the proportion of questions the system attempts.

The 2024 study Hallucination-Free? assessed specialist legal research tools. It reported hallucinations between 17% and 33% in the tested products. The result concerns those systems and legal tasks at the time of assessment.

Those figures are not an error estimate for Australian tax advice. A general-knowledge benchmark is also no substitute for testing questions about Australian legislation and the client facts that change its application.

Tax questions depend on the income year and the version of the law. The treatment can also change with the entity or transaction. A benchmark about general facts does not reproduce those conditions.

Use published studies to understand failure modes. For a practice decision, establish whether the tool supplies sources you can inspect and whether staff can review each consequential claim before use.

Professional Reports With AI Errors

Published professional reports have contained nonexistent references and false quotations, with correction or withdrawal following. Two reported cases show different consequences.

In an Australian government assurance review, Associated Press reported a fabricated court quotation and references to nonexistent research. The department said the supplier agreed to repay the final contract instalment. A revised report disclosed generative AI use.

The Australian report coverage distinguishes the reference errors from the report’s recommendations, which the department said remained unchanged. The department’s report page also records a later correction in 2026.

In Canada, a professional-services firm withdrew its cybersecurity study about loyalty systems after apparent AI hallucinations and false footnotes were identified. Reporting on the withdrawal says the firm was reviewing how the study reached publication.

The Canadian study was marketing material, separate from client project work. The response reported in the Indian Express made that distinction. It would be inaccurate to describe both incidents as invented tax advice delivered to clients.

A fabricated reference does not establish that every conclusion in a report is false. It does mean the reviewer must establish which claims depended on that reference and reassess those claims.

Checks That Catch Hallucinations

Review AI output by checking each claim that could change the client’s advice or obligations. A second fluent answer from the same model does not prove that the first answer is correct.

1. Identify what the draft relies on. Mark legal propositions and citations. List every rate or threshold used in a calculation, and every deadline the client will act on.
2. Open each original source independently. Find the ruling in the ATO Legal Database or the judgment through the court or a recognised legal database. Confirm the document exists and supports the exact claim.
3. Check the applicable version. Establish the income year and the client facts. Read amendments, withdrawal notices and limits on the source’s application before carrying its conclusion into advice.
4. Recheck figures against the ATO. Match rates and thresholds to the relevant year. Confirm the client’s lodgement and payment dates from the applicable ATO records, then recalculate using the checked inputs.
5. Compare the conclusion with the evidence. A genuine link can lead to a page that says something different. Read the relevant paragraph and its surrounding qualifications.
6. Record who reviewed the work. Keep the output version, source references and material corrections with the reviewer’s name and sign-off. Follow the practice’s retention rules for the file.

The TPB’s AI guidance says practitioners remain responsible for reviewing and verifying AI-generated content before relying on it. Its record-keeping guidance calls for evidence of that review.

When a citation cannot be found, hold the dependent advice while a qualified reviewer resolves the point from genuine authority. Asking the model for another reference can help locate a lead, but that replacement needs the same source check.

For a fictional deduction draft, the review succeeds when each authority is located and the conclusion follows from its actual reasoning. The file must show the checked source and the person who approved the resulting advice.