The Distrust Machine
Emma Roth at The Verge called it “a new era of distrust”. Right name. AI detectors — GPTZero, Pangram, Turnitin’s built-in — are being used by teachers and publishers to decide who’s honest, and the distrust isn’t a side effect. It’s the product.
Old-school plagiarism checkers compared your writing against a database of everything ever published. Matching text was a fact you could verify. The new detectors don’t work that way. They run your words through their own AI model and ask it to guess whether a human wrote them. A guess, with a confidence score, sold as a verdict.
And it’s being used as a verdict everywhere. A Center for Democracy and Technology survey found 43 percent of US middle and high school teachers regularly ran student work through these tools in 2024–25. Turnitin switched AI detection on by default for universities that never asked for it. Publishers screen manuscripts. Employers screen applications.
The problem isn’t that detectors are wrong sometimes. It’s the shape of their wrongness. When Stanford researchers tested seven detectors on essays by non-native English speakers, the tools flagged honest human writing as AI-generated at rates up to 61 percent — because “sounds like AI” is suspiciously close to “grammatically careful, slightly formal, written by someone whose first language isn’t English.” The Markup documented international students hauled in for cheating on work they demonstrably wrote themselves. OpenAI built its own detector, admitted it caught only about a quarter of AI text, and quietly killed it in July 2023. The company that makes the machines couldn’t build a lie detector for them. Teachers are expected to do better with a dashboard.
Now the counterarguments, because they’re real.
First: cheating is real, and teachers need something. Yes. Students do paste ChatGPT output into essays, and pretending otherwise is its own dishonesty. But the tool has to fit the crime — and a false positive in an academic dishonesty case isn’t a rounding error, it’s a kid’s semester and record. Turnitin disputes the Stanford numbers and claims under one percent false positives in its own testing, while conceding that shorter documents and non-native writing get flagged more often. Even taking the vendor’s best case: when your instrument is least reliable exactly for the students least equipped to fight an accusation, you’re not enforcing integrity. You’re taxing it.
Second: the detectors will get better, watermarking is coming. I hope so — and I’ll change my mind the day provenance replaces probability. Watermarking isn’t a guess; it’s a fingerprint baked into the output, verifiable after the fact. That’s a real fix. But it only works if every major lab cooperates, if the marks survive editing and translation, and if the “humanizer” services that exist specifically to scrub AI tells don’t stay a step ahead. Until then, we’re not doing detection. We’re doing divination with a progress bar.
Third, and the one that scares me most: something is better than nothing. No. A bad instrument in a high-stakes setting isn’t neutral. It manufactures false accusations, and it teaches everyone to game it. Students learn to write blander so they pass the bot-check. Teachers learn to trust a score over a kid’s actual work. The detector doesn’t catch more cheaters — it makes honest writing look more like the thing we’re trying to catch. That’s the doom loop: the more we distrust, the more machine-like honest writing becomes, the harder the problem gets.
The fix isn’t a better detector. It’s better assignments — drafts, in-class writing, oral defenses, work that can’t be pasted. It’s treating a flag as the start of a conversation, not the end of one. And it’s remembering that the people these tools are pointed at are mostly kids who did the work.
The machines got good at imitating us fast. The detectors got good at imitating judgment. Only one of those is worth keeping.
Sources: The Verge — “AI detectors are creating a new era of distrust” · CDT teacher survey · The Markup on Stanford’s detector-bias research · OpenAI retires its AI classifier · via The Brutalist Report