Someone posted their great-great-grandfather's Civil War story in the comments of a genealogy YouTube channel a few weeks ago. He'd fought with the 14th Virginia Cavalry. He was at Gettysburg. And according to ChatGPT, he was captured there.
He wasn't. ChatGPT had stitched two true facts together (the regiment fought at Gettysburg, and plenty of soldiers were captured that week) into a story that sounded right and wasn't.
The invented capture.
The real record had been sitting in the National Archives the whole time. His Compiled Service Record, digitized on Fold3, shows what actually happened: he wasn't captured until later, at a different battle, and he died two weeks before Appomattox. Point Lookout's prisoner rolls back it up. The true story didn't need embellishing. It just needed someone to check a primary source instead of asking a chatbot to guess.
Why the guess sounded so confident.
A large language model doesn't retrieve facts. It predicts the next plausible word, based on patterns in whatever it read during training. Ask one about a Civil War regiment and it has seen thousands of accounts of soldiers being captured, wounded, or killed near Gettysburg that week. It knows what a plausible sentence about a captured soldier sounds like. What it can't do is check whether this specific man, on this specific date, was actually one of them. Confidence and accuracy are not the same thing, and a general-purpose chatbot has no mechanism for telling you which one you're getting.
The two Henrik Larsens.
The Gettysburg mistake was one flavor of the problem. A different one showed up not long after in a Norwegian genealogy Facebook group, and this one had nothing to do with a battle. A member asked ChatGPT to summarize her great-great-grandfather Henrik Larsen's emigration from Bergen, and it answered with real specificity: a ship name and a departure date in 1881, plus a homestead in Wisconsin the year after. About half of it was true. The rest belonged to a second Henrik Larsen, born in the same decade in the same parish, who emigrated twelve years later and settled two states away. The model had pulled both men's traces from whatever text it had encountered mentioning the name Henrik Larsen and the port of Bergen, and combined them into one continuous life, because nothing in its training ever told it these were two different people. She almost added the merged paragraph straight to her tree. What actually separated the two men was a single passenger manifest from 1893, listing the second Henrik Larsen's exact age, his wife's name, and a destination that matched nothing from the first man's life. One page did what the chatbot's fluent paragraph couldn't: told the two Henriks apart. A common name in a small parish is exactly the condition where that kind of merge happens, and the paragraph reads as one clean biography, with nothing in it warning a reader that two lives just became one.
One wrong sentence, ten thousand family trees.
Genealogy carries a risk that most AI use cases don't share, and the reason is specific, not abstract. Ask a chatbot to help revise a paragraph of a novel, and a bad sentence dies with that draft. Ask it about a specific ancestor, and a wrong immigration date or a merged identity gets typed into a tree. That tree gets shared with a cousin. The cousin exports it as a GEDCOM file and imports it into their own tree, since most tree software has no mechanism that stops an unsourced fact from crossing platforms. A decade later, a DNA match looks at the same invented ancestor sitting on three or four cousins' trees and reads the agreement as confirmation, on the theory that several people can't all be wrong about the same thing. They can be. They copied the same guess. Worse, there's no clean way to undo it once it's out. Deleting the error from your own tree doesn't touch the copies already sitting on a cousin's account, and nothing forces anyone to notice a correction even after you point it out directly. A wrong sentence in a novel stays exactly as wrong as it was the day it was written, read by however many people pick up the book. A wrong ancestor gets inherited: copied, exported, re-imported, and treated as settled fact by people who never saw the chatbot conversation that invented him in the first place.
What a chatbot is actually good for.
A general-purpose chatbot still has a place in a research workflow. The job has to match what it's actually built to do. Handed a photograph of a Cyrillic parish register, a chatbot is a genuinely fast first pass at translation, better than nothing and quicker than working through a dictionary letter by letter. Asked what a Confederate Compiled Service Record actually contains, or why no county-level 1890 census exists, it explains the record type correctly, because that's stable, well-documented information repeated the same way across most of what it read during training. It's also a reasonable place to ask how DNA segment lengths roughly translate into a relationship range, since that's a similarly stable, well-established concept explained the same way thousands of times over. Stable questions get stable answers. Handed a document you've already found, a fifteen-page probate file, say, it can summarize what's on the page faster than you can read it twice. All three jobs share one property: the chatbot is working with something already placed in front of it, a document to read or a fact to explain. Change the question and the ground shifts. The moment it becomes what happened to this specific, unverified person, there's nothing in front of the chatbot to check against. It answers anyway, because answering is the one thing it always does, and that's the entire gap between translating a page and inventing one.
Where AI actually earns its keep.
None of this makes AI useless for genealogy. It means most people are pointing it at the wrong kind of problem. We've watched this exact split show up in genealogy forums: AI is genuinely good at constraint logic, working out that three DNA matches can't all be second cousins once you know their shared segment lengths. It's genuinely bad at biography, inventing what probably happened to someone based on what usually happens to people like them. One is arithmetic. The other is a guess wearing a costume. A shared segment length is a number you can check against another number. A life story is not, and treating the two as the same kind of problem is where most chatbot-assisted genealogy goes wrong. The same arithmetic strength shows up in catching an impossible timeline: a marriage record that would require a six-year-old bride, or a father who died three years before a child was born. Those are contradictions a system can catch by checking two dates against each other, not by understanding who anyone actually was.
The professional's version of the same question.
Serious researchers ask a version of this question constantly, and it deserves a straight answer instead of a shrug: why not just use ChatGPT for everything? Because analyzing a document and closing a research loop are two different jobs, and only one of them ends with something you can actually trust on a tree. A chatbot can read a document you hand it and tell you what it says. It cannot go find that document in the first place, check it against three other records that might contradict it, and propose a specific, sourced change to your tree that you then get to accept or reject. That loop, from a question to a retrieved record to a proposed edit a person actually reviews, is the part general-purpose chat was never built to do. It's also the part that turns one research session into something that still holds up on a second look six months later. A professional who already pays for a records subscription isn't looking to replace it with a chatbot. They're asking whether the chatbot can do the one thing that subscription still doesn't: check a claim against a document before it gets published as fact.
Retrieval instead of recall.
The technical fix for all of this has a name: retrieval-augmented generation, and the distinction is worth understanding because it explains why two systems can answer the same question with such different failure rates. A bare chatbot answers from what got compressed into it during training: billions of parameters shaped by patterns in text, with no live connection to any particular document at the moment it writes a sentence. A retrieval-based system works differently. It takes the question, searches an actual index of records (census pages, draft cards, ships' manifests), and pulls back the specific documents that might answer it before writing a single word of response. In practice, that lookup step works by comparing the question against an index built from the records themselves, not from someone's summary of them, which is what lets a system point at page fourteen of one specific file instead of a general impression of what records like it tend to say. Then it's restricted to sentences it can point back to one of those documents. That's not a tuning knob on the same machine. "Predict the next plausible word" and "find the record, then describe only what's on it" are two different jobs. One treats a document as raw material for a guess. The other treats it as the only thing allowed to produce a claim. Making the underlying model bigger doesn't fix this by itself, since a larger model is still predicting text, just with a sharper sense of what plausible text looks like. The fix has to sit at the architecture level, in how the system decides what it's allowed to say before it ever starts writing a sentence. When the retrieval step turns up nothing, the honest output is nothing: not a well-written paragraph built to fill the space where a record should have been.
Records first, narrative second.
That split is the whole design principle behind AncestorIQ's Deep Research. Every claim has to point back to an actual record: a census page, a draft card. Not a well-written paragraph. If the record doesn't exist yet, the honest answer is "we don't know", not a confident guess wearing period detail. It's slower than asking a chatbot. It's also the only version of the answer you can actually build a tree on.
The Confederate soldier's real story never needed a chatbot's help. It just needed someone to look.
