How Accurate Are AI Detectors, Really? The Numbers Nobody Advertises

AI detectors have flagged the US Constitution as machine-written. They've flagged Genesis. They've flagged essays by students who'd never touched a chatbot — while waving through AI text that someone spent ten minutes paraphrasing. Both failures happen for the same reason, and it's built into how these tools work.

I've tested these tools extensively for this site. Here's how they actually score text, what the accuracy claims hide, and what to do if one accuses you.

A confident percentage is not the same thing as a correct one.

How detectors actually work: perplexity and burstiness

Strip away the marketing and most detectors measure two things. Perplexity asks: how surprised would a language model be by each next word? AI text is generated by picking probable words, so it's low-perplexity — smooth, predictable, statistically average. Burstiness asks: how much does sentence rhythm vary? Humans write a five-word punch, then a rambling forty-word thought. AI tends toward even, medium-length sentences unless pushed.

Low perplexity plus low burstiness equals an AI flag. Notice what's missing: any actual knowledge of where the text came from. It's statistical vibes, formalized. That's why the Constitution gets flagged — it's quoted so often in training data that every word is highly predictable.

The accuracy shell game

When a vendor says "99% accurate," ask: on what? Pure, unedited output from one model, tested in-house, usually. Change any variable — a newer model, a human edit pass, a non-native English writer, text under 300 words — and the numbers move, sometimes a lot. Independent studies have repeatedly found false positive rates in the single digits for the better tools and much worse for the weaker ones. Even 1-2% is brutal at scale: run 5,000 student essays through a detector and a 2% false positive rate accuses roughly 100 innocent people. Per assignment.

It says a lot that OpenAI shut down its own text detector back in 2023 for low accuracy, and that major plagiarism-tool vendors publicly caution against using scores as sole evidence. The people closest to the technology are the most careful about it.

Who gets falsely flagged (the ugly pattern)

A widely cited Stanford study found detectors flagged essays by non-native English speakers as AI-generated at rates above 50% in some configurations, while near-perfectly clearing native speakers on the same task. The mechanism is obvious once you see it: writing in a second language tends toward safer, more standard constructions. Lower perplexity. More machine-like, statistically.

The same logic hits technical writers, anyone trained in rigid five-paragraph essay structure, and frankly anyone who writes cleanly. The tools punish polish. Meanwhile, AI text that's been paraphrased — by hand or by tool — routinely drops from "98% AI" to "mostly human" in minutes. So the system over-catches innocent careful writers and under-catches motivated cheaters. That's not a bug being fixed; it's the shape of the method.

What this means for students and writers

Students: your best protection is process, not prose. Write in a tool with version history, keep your outline, and save sources. If you use AI legitimately — brainstorming, feedback on a draft — know your institution's actual written policy, because "allowed with disclosure" and "forbidden" both exist in 2026, sometimes at the same school.

Writers and freelancers: clients increasingly run detectors, wrongly treating them as truth machines. Protect yourself the same way — drafts and history — and make your writing sound like a person, which is good advice regardless of robots. We've broken down the vocabulary tells in AI words to avoid and the full editing approach in how to humanize AI text. And if you're wondering whether search engines run detectors on your content: that's a different question with a different answer, covered in does Google penalize AI content.

My honest take

Detectors aren't useless — as one weak signal among many, they can prompt a human conversation. Used as verdicts, they're a coin with a thumb on it, and the thumb presses hardest on people writing in their second language. Institutions that expel students on a percentage from a black box aren't practicing integrity. They're outsourcing it.

Frequently asked questions

The uncomfortable summary: we built tools that guess, marketed them as tools that know, and handed them power over grades and careers. Until that changes, keep your drafts. They're worth more than any percentage.