Don’t count on an AI chatbot alone to judge illness or legal questions; before seeing a doctor, going to court, or spending a lot, find the sources it cites and check the original

Handed a full case, AI names the disease over nine times in ten. But when ordinary people ask it about symptoms, they recognize the disease less often than people who search the web themselves, and whether to go to emergency is not judged more accurately. On legal questions, general-purpose AI answers have fabricated content at least 58% of the time, and they will follow wrong premises in your question.

Cost

No money. Each check takes a few minutes to half an hour. Have the AI write the statute, guideline, or paper it relies on; open the original yourself and find that sentence. The habit to change is not acting just because it speaks with certainty.

Benefit

First medicine. A 2026 Nature Medicine randomized trial enrolled 1298 ordinary people in the UK; the protocol was preregistered. Each got two prepared cases and had to judge the disease and whether to go to emergency or watch at home. Some used an AI chatbot — GPT-4o, Llama 3, or Command R+. Controls used their usual methods, mostly web search or the NHS site. Handed a full case directly, AI named a relevant disease 94.9% of the time and picked the right care setting 56.3% of the time. Ordinary people using the same AI named a relevant disease under 34.5% of the time and picked the right setting under 44.2%. Controls recognized a relevant disease 1.76 times as often as AI users (95% CI 1.45–2.13, credible range). Recognizing red-flag serious illness was 1.57 times (1.28–1.92). Judging where to go was similar on both sides. The authors reviewed chat logs; problems were two: users did not tell all symptoms, and when AI named the right disease users did not take it. Next law. A U.S. research team in 2024 asked public models including ChatGPT 4 about U.S. federal case law. ChatGPT 4 had fabricated content in 58% of answers; Llama 2 up to 88%. Fabrication means content that does not match legal fact; the paper calls it “hallucination.” Models cannot tell which of their own lines are made up. When the user’s question carries a wrong legal premise, they often accept it wholesale. The same team in 2025 tested professional legal AI tools for lawyers. Tools that claim not to fabricate still had fabrication in 17% to 33% of answers. It looks worthwhile because AI is always on, free, and speaks with a doctor’s or lawyer’s certainty.

Notes

Controversy. A 2023 study drew 195 patient questions from a U.S. forum and had clinicians score without seeing the source. 78.6% of scores rated ChatGPT’s answer better than a doctor’s. But that compared how well the answer was written, not whether the asker’s final judgment was right. The 2026 trial used 2024 models; newer ones may be stronger. Yet the same trial also found that AI’s solo exam score does not predict how ordinary people do with it. Why AI fabricates: OpenAI researchers explained in a 2025 non-peer-reviewed paper that training and scoring reward “guessing an answer”; saying “I don’t know” scores nothing, so when unsure it still speaks with certainty. The check method: have it write the statute name and article number, or the guideline or paper title; open the original on an official site and find that sentence; if you cannot find it, treat it as unsaid. When asking, do not put your conclusion in the question — ask “what does the law say in this situation,” not “have I already won.” Sudden face droop, one-sided weakness, crushing chest pain: do not ask AI first; call 120. See Section 13, Item 3 (face droop, one arm weak) and Section 13, Item 7 (crushing chest pain). No money for a lawyer: Section 7, Item 3 (apply for legal aid). Benefit set to small because only intermediate markers like “disease recognized or not” exist, with no death-reduction numbers. Beneficiaries are you and family.

Sources · 5

1. Bean AM, Payne RE, Parsons G, et al. (2026). Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine, 32(2), 609–615. doi.org/10.1038/s41591-025-04074-y
2. Dahl M, Magesh V, Suzgun M, Ho DE (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis, 16(1), 64–93. doi.org/10.1093/jla/laae003
3. Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE (2025). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies, 22(2), 216–242. doi.org/10.1111/jels.12413
4. Kalai AT, Nachum O, Vempala SS, Zhang E (2025). Why Language Models Hallucinate. arXiv:2509.04664（预印本，备注里那项）. arxiv.org/abs/2509.04664
5. 反方：Ayers JW, Poliak A, Dredze M, et al. (2023). Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, 183(6), 589. doi.org/10.1001/jamainternmed.2023.1838