Protocol
Run your own detector test
Twenty texts, one afternoon, two numbers that tell you whether to trust a detector on your writing.
01 · Setup
The 20 texts
02 · Steps
The protocol
Collect 10 human texts
Writing like the writing you will check: same genre, 300 to 800 words. Use work written before November 2022 where you can, so no AI was involved.
Set H · 10 texts · label: humanMake 10 AI texts on the same topics
Five straight from a model, three you edit for ten minutes, two run through a paraphraser. Same topics stop the test measuring subject instead of authorship.
Set A 5 · Set E 3 · Set P 2 · label: AIFix your rule before you look
Decide what counts as 'called AI': the tool's own AI label, or a score above a number you write down now.
rule: called_ai = tool label == 'AI' (or score >= your threshold)Shuffle, then run each text once
Paste each text in a fresh session in shuffled order. Record the score and label exactly as shown, plus the passages it flags.
One run per text. No re-runs to 'see if it changes'.Count the two error rates
False-positive rate is the share of your human texts it called AI. Detection rate is the share of AI texts it called AI.
fp_rate = human_called_ai / 10 · detection = ai_called_ai / 10Compare on false positives first
A tool that accuses your human writers is the costlier mistake. Between tools with the same false-positive rate, take the higher detection rate.
Re-run the same 20 texts whenever the vendor ships a new model.
03 · Record
Your results sheet
Paste into any spreadsheet. One row per text, one sheet per detector.
id,set,truth,words,tool,score,label,called_ai,flagged_passages h01,H,human,612,,,,, h02,H,human,455,,,,, a01,A,ai,580,,,,, e01,E,ai,601,,,,, p01,P,ai,540,,,,,
04 · Count
Work out the two rates
Detectors change their models