How AI Detectors Work (and Where They Fail)

AI detectors are sold with confident numbers, like "99% accurate". The reality is messier. Knowing how they work tells you how much to trust a score, and what kind of writing gets wrongly flagged.

The three main approaches

Almost every detector on the market uses one or more of these methods.

1. Statistical scoring

The detector runs your text through a language model and measures how predictable each word was. That's perplexity and burstiness. A research method called DetectGPT, published by a Stanford team in 2023, pushed this further: it makes small rewrites of your text and checks whether the original sits at a "peak" of likelihood the way AI output tends to.

2. Trained classifiers

Here a model is trained on thousands of examples labeled "human" and "AI", and it learns to tell them apart. Most commercial tools, including the ones schools use, work this way or mix it with statistical scoring. They're only as good as their training examples. A classifier trained mostly on one chatbot's output can miss text from a newer one.

3. Watermarks

Some AI companies can hide a pattern in the words their model picks, invisible to readers but detectable with the right key. Google released its SynthID text watermarking as open source in October 2024. A watermark only works for text from that company's model, and heavy editing weakens it.

How accurate are they?

Less than their marketing suggests. Two cases are worth knowing:

• OpenAI's own detector. OpenAI launched an AI text classifier in January 2023. By its own figures it caught 26% of AI-written text and wrongly flagged 9% of human text. The company withdrew it in July 2023, citing its low accuracy.
• The Stanford study on non-native writers. In 2023, Liang and colleagues ran essays from the TOEFL English exam through seven popular detectors. On average, the detectors labeled more than half of these human-written essays as AI-generated. Essays by US eighth-graders were judged far more fairly.

Detectors have improved since then, and some vendors publish low false-positive rates on their own test sets. But a small error rate still means a lot of wrong accusations when millions of essays are checked every term. Lesson 6.2 explains how to read a detector score with that in mind.

Whose writing gets wrongly flagged

Detectors look for predictable, even text. So they're most likely to misjudge people who write that way for good reasons:

• People writing in a second language, who stick to words and patterns they're sure of.
• Students trained in rigid formats, such as the five-paragraph essay.
• Technical, legal and scientific writers, whose fields reward plain, standard phrasing.
• Anyone who ran their own writing through a grammar tool that smoothed it out.
• Very short texts, where there isn't enough material to judge.

If this is you, read Falsely flagged as AI? What to do and start keeping version history now, before you need it.

What this means for you

A detector score is a guess about style. It isn't a fact about who wrote something. That cuts both ways. It means you shouldn't panic over one high score on your own work. It also means that "beating" one detector proves little, because the next tool may read your text differently.

The approach that holds up across every detector is the one this course teaches: write with real detail, real opinions and a natural rhythm. Lesson 1.4 lists the 12 patterns detectors and readers pick up on most.