Why ChatGPT can't actually handle your health — from blood test to bedside.

This isn't an attack on language models

Lipa uses language models too. We use them at the very end of our analysis pipeline to compose the plain-English explanations our members read. We have nothing against ChatGPT or Claude or Gemini — they are remarkable tools and they have made many things possible that weren't a few years ago.

This article is honest about what they can and can't do for the questions that actually matter in health: interpreting your blood test, weighing a medication change, reading a discharge summary, decoding an imaging report, judging whether a symptom is benign or worth pushing your doctor on. Bloodwork is the canonical example — it's what we built Lipa around — but the failure modes documented below apply to every high-stakes health conversation, not just lab values. We use these tools ourselves. We also know what they can't do alone.

The "AI scored 92% on diagnosis" study, in context

You may have seen the headline. In November 2024, a study published in JAMA Network Open by Goh and colleagues randomized 50 physicians to use either GPT-4 or conventional resources for diagnostic reasoning on a set of clinical case vignettes[1]. The widely-shared finding: GPT-4 alone scored about 92% on the diagnostic reasoning task, while physicians using GPT-4 scored 76%, and physicians using conventional resources scored 74%. The model alone outperformed clinicians using the same model.

It's a real result. It deserves to be taken seriously. But it also deserves to be contextualized.

What the study tested: diagnostic reasoning on standardized clinical vignettes. Cases were curated, well-formatted, and designed to test differential diagnosis capability. Researchers gave GPT-4 clean, structured inputs and graded its answers against gold-standard diagnoses.

What the study didn't test:

• Real-world blood test interpretation with messy lab PDFs and missing context
• Whether the model's cited reasoning relied on real or hallucinated sources
• Longitudinal patient data — your prior tests, your trends, your history
• Whether the answers would change with the latest research from the past 6 months
• Funding-source awareness in cited studies
• Removal of retracted papers from the model's reasoning
• The complete patient context: medications, lifestyle, family history, current symptoms

In other words: a controlled benchmark on clean inputs is not the same as reading a real patient's blood test. The 92% number is impressive and important — but it tells you what GPT-4 can do in a sterile environment, not what it does with your hs-CRP at 2.8 mg/L on a PDF you took with your phone in bad lighting.

The hallucination problem

Language models work by predicting plausible next tokens, not by retrieving facts from a database. When you ask ChatGPT to support a claim with a citation, the model generates something that looks like a citation: an author name, a journal title, a year, a PMID. These are all statistically reasonable guesses based on the model's training data — but they are not lookups against a real database.

The result is the well-documented hallucination problem. A 2023 ACM survey of hallucination in language generation systems catalogued the failure modes across natural language tasks[2]. In the medical domain, multiple studies have specifically tested LLM citation generation and found that a substantial fraction of cited references — often more than 30 percent — are fabricated[3][4]. The model produces plausible-sounding PMIDs that don't exist, real PMIDs assigned to wrong papers, real papers with the wrong authors, or real authors paired with fabricated journal titles.

For someone trying to understand their blood test, this is a fatal flaw. If you ask ChatGPT "what does the research say about my hs-CRP being 2.8 mg/L?" it will confidently give you an answer with citations. The answer might be correct. The citations might be real. Or the citations might be invented. You have no way to tell without checking each one against PubMed manually — and at that point, you've done the work yourself.

This is the single biggest reason general LLMs are not appropriate for blood test interpretation as a standalone tool. The user has no signal to distinguish a confidently-stated correct answer from a confidently-stated wrong one.

The frozen knowledge problem

Language models have a knowledge cutoff. The training data is collected up to a certain date, the model is trained, and then released. Once deployed, the model knows nothing that happened after its cutoff. ChatGPT, Claude, and Gemini all have cutoffs — typically months to over a year before the time you're using them.

This matters for blood test interpretation because medical research moves. A 2026 systematic review and meta-analysis on omega-3 supplementation and hs-CRP — the kind of paper that would reasonably change your understanding of an intervention — won't be in a model whose cutoff was 2024. Your "AI" interpretation will rely on the literature available before that systematic review existed, even though the new evidence is the more reliable answer.

This isn't fixable by clever prompting. The model literally does not have the data. It will confidently summarize an older state of the literature and present it as the current consensus.

The retraction problem

Worse: language models also can't remove papers that have been retracted. When a study turns out to be fraudulent, methodologically broken, or simply wrong, journals issue retraction notices. Reputable databases like PubMed flag retracted papers. Real medical research workflows exclude them.

A language model trained on text that included a retracted paper has no mechanism to "unlearn" that paper. The retracted findings remain encoded in the model weights, and the model will continue to cite them as if they were current evidence. There is no patch. There is no recall.

This is a documented problem in the LLM-in-medicine literature[5]. For blood test interpretation, where the wrong answer can lead to wrong decisions, the inability to remove retracted papers is a serious limitation.

The memory problem

ChatGPT does not remember your previous tests. Each conversation is largely a fresh start. The latest versions have introduced limited memory features, but these are not designed to track longitudinal patient data — they are designed to remember conversational preferences.

For blood test interpretation, longitudinal data is often the most informative signal. Your hs-CRP at 2.8 mg/L is one data point. Your hs-CRP at 2.8 mg/L following 1.4 mg/L six months ago is a trajectory. Your hs-CRP at 2.8 mg/L following 1.4 mg/L six months ago following 4.0 mg/L two years ago is a story. The trajectory is the medicine.

A general AI tool with no persistent memory of your prior tests can only see the snapshot. It can't see the trend. It can't compare to your baseline. It can't notice that your fasting insulin has been creeping up for 18 months while your fasting glucose has stayed normal — the kind of subclinical pattern that early-warning matters for.

The funding-source problem

Most cited research is influenced — sometimes obviously, sometimes subtly — by who paid for it. Industry-funded studies have a well-documented tendency to produce results favorable to the funder[6][7]. Independent academic studies and government-funded research are not free of bias either, but the structural incentives are different.

A serious medical research synthesis weights independent and government-funded research more heavily than industry-sponsored research, and at minimum makes the funding source visible so the reader can judge for themselves. This is standard practice in evidence-based medicine and in any well-conducted systematic review[8].

Language models don't track funding sources. When ChatGPT cites a paper (or hallucinates one), it doesn't tell you who funded the underlying research. You can ask, and the model will confidently tell you something — but the answer is generated from training-data statistics, not from a database lookup. There is no audit trail.

The "no doctors will use it" problem (which is also a market gap)

OpenAI's usage policies explicitly prohibit using ChatGPT to provide unsupervised medical diagnosis[9]. Anthropic's policies state that Claude is not designed to provide medical advice[10]. Google's Gemini policies are similar. These are not minor disclaimers — they are deliberate liability decisions by companies with very large legal exposure.

What this means in practice: the major AI companies will not market their models as medical tools. They will not run a "Use ChatGPT for your blood test" advertising campaign. They will not optimize their systems for clinical reliability. They will continue to be excellent general-purpose tools, and they will continue to leave the medical interpretation space to specialized products.

The space is open by design. The question is what you build in it.

What ChatGPT is actually good at, when it comes to your health

This article isn't a takedown. Language models are remarkable tools, and they have legitimate uses for someone trying to understand their own biology:

What we built instead, and why

Lipa is what you get when you take everything language models are good at and combine them with everything they're not. The model is the ingredient. The product is what you build around it.

Specifically, Lipa wraps the language model with five things ChatGPT alone doesn't have:

1. A curated, real research corpus. We ingest peer-reviewed studies daily from PubMed, Europe PMC, Cochrane, ClinicalTrials.gov, bioRxiv, medRxiv, and major clinical guideline bodies. The corpus is real. Every paper has a real PMID. The model retrieves from this corpus rather than generating from training-data memory.
2. Citation grounding with no fabrication. Every claim Lipa generates is forced through a grounding step that ties it back to a specific retrieved paper. If the model wants to cite something, it has to cite something from the retrieved set. Hallucinated PMIDs are structurally impossible.
3. Daily corpus refresh + retraction handling. When a new paper publishes, we ingest it within 24 hours. When a paper is retracted, we remove it from the retrieval pool within hours. The corpus stays current.
4. Funding-source tracking. Every paper in our corpus is tagged with its funding source — independent academic, government, non-profit, industry, mixed, or undeclared. Independent research is weighted higher in our grading. The funding label is visible on every insight you see.
5. Persistent memory of your biology. Every test you upload is saved to a vault we own and you control. Year-over-year trends, cross-marker patterns, longitudinal trajectories — these all become visible only when multiple tests are read together. The trajectory is the medicine.

None of these are individually magic. They are deliberate engineering decisions to address specific failure modes of using language models alone.

The bigger point

The fact that GPT-4 can score 92% on a diagnostic reasoning vignette is meaningful. It changes what specialized tools have to do to add value. It means that thin wrappers around an LLM API — products that just send your blood test to ChatGPT and reformat the response — are not differentiated. Anyone can build that. The model does most of the work.

But the model alone is not enough for any task that requires real, verifiable evidence, longitudinal context, and rigorous research handling. For blood test interpretation specifically, the moat is the layer around the model: the curated corpus, the retrieval, the grading, the funding tracking, the persistent vault, the personalized matching. That's where Lipa lives. That's why we built it.

We use language models. We don't pretend they do everything.