How accurate are AI-written exam questions? What real jobs show
How often are they wrong? Of the 40,257 questions the models wrote here from October 7, 2026 to November 5, 2026, 23.2% were removed by the rules on how a question is written and whether its quotes support the key. Counted against all questions written, 10.7% quoted a line that is not on the page they cite and 7.0% got a different answer from a second model: added together, at least 17.7% were unsupported or disputed. 60.5% were thrown out for all reasons, and none of those reach a student. The checks cannot judge the medicine, so a question that passes can still be wrong, and that rate is not measured here.
Every question this site writes is checked before a student sees it, and the checks keep a count. This is what 30 days of real jobs on medical textbooks, lecture decks and notes show, in a count taken once a week: how many questions a model writes that cannot be traced to the page, that a second model answers differently, or that repeat one already written.
The numbers, counted from the call log
| What the checks counted | Share of questions written |
|---|---|
| Questions the models wrote | 40,257 in 11,068 calls |
| Kept, every check passed | 39.5% |
| Thrown out, all reasons | 60.5% |
| Quote not found on the cited page | 10.7% |
| Second model answered differently | 7.0% |
| Repeat of a question already written | 18.5% |
| Writing or grounding flaw (all of the above, a vague lead-in, a key the quotes do not support) | 23.2% |
Counted from the call log on November 6, 2026, over October 7, 2026 to November 5, 2026, and counted again every Friday.
What each rejection means
Quote not found on the cited page. Every question has to carry one or two sentences copied out of the document, with the page they sit on. The server looks the sentence up in the stored text of that page and of the pages either side of it, word for word or as a near match that keeps its numbers and negations. If it is on none of them, the model wrote a question about something the material does not say, or misremembered the wording, and the question is dropped. This is the check that catches an invented fact.
Second model answered differently. A question with options goes to a second model with its quotes and never with the answer key. If it cannot arrive at the keyed answer from the quotes alone, the quotes do not prove the answer, and the question is dropped. This is the check that catches a right-sounding question with the wrong key, or a key the quote does not support.
The second model answers only questions with options to choose from: a short answer question has none, so it skips this check and is graded against the source when you answer it. A question the second model does not answer in time is kept on the quote check alone, with a badge that says so, and is checked again later; if the second model then disagrees, the question is taken out of the exam. The second model is from a different family whenever one answers; when none does, a model of the same family does the check, and as a last resort the model that wrote the question.
Repeat of a question already written. Each new question is compared with everything already generated from the document. A near repeat is dropped, so a long run gives different questions rather than the same facts restated.
Writing or grounding flaw. "All of the above", "none of the above", a stem that asks which statement is true, a correct option more than 150 percent and more than 50 characters longer than the longest distractor, a question that refers to "the text": these are the tells examiners are trained to remove, and the writing rules remove them here. The grounding rules drop a key the quotes do not support, a topic the source does not cover, a question in the wrong language and a weak explanation.
Why a generator that shows no quote cannot tell you its accuracy
A tool that hands you a quiz without the sentence each question came from has no way to count how many of its questions were unsupported, because nothing in its pipeline looked. Its accuracy is whatever you find when you check the questions yourself, which is the work you were trying to hand off. One competing tool's own help page asks its users to review a generated quiz before relying on it, which is the honest thing to say when nothing was checked. The numbers above exist because the checking happens before the student sees anything, and because what was thrown out was counted rather than quietly dropped.
What published studies found
The counts at the top of this page come from automatic checks that throw a question out before anyone sees it. Researchers have also measured AI-written medical questions by hand, with experts or students reading every item. Four peer-reviewed studies are summarized below, each as the paper reports its own results. All four looked at OpenAI models available in 2023, so they describe those models at that time.
Pathology: 60 questions written from the authors' own text
Ngo A, Gupta S, Perrine O, Reddy R, Ershadi S, Remick D. ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions. Academic Pathology, 2023.
A pathology teaching team gave ChatGPT text they had written themselves and asked for 60 multiple choice questions, each with an explanation of the right answer and of the wrong ones. Only 19 of the 60, or 32 percent, came back as correct questions with correct answers and explanations. A further 25 percent had answers that were wrong or misleading, and in many cases the explanation of the wrong options was missing.
Neurophysiology: 161 students answered both sets
Laupichler MC, Rother JF, Grunwald Kadow IC, Ahmadi S, Raupach T. Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions. Academic Medicine, 2024.
An experienced medical educator wrote 25 questions with five options each, and ChatGPT wrote 25 more to the same learning objectives. Students took both sets in a practice test before their neurophysiology exam, and the final analysis held 161 students, 25 human questions and 21 ChatGPT questions. The difference in difficulty between the two sets was not statistically significant, but the human questions separated stronger from weaker students better, with a mean discrimination of 0.36 against 0.24. Asked who had written each question, the students were right 57 percent of the time.
Medicine and surgery: 100 questions scored by five assessors
Cheung BHH, Lau GKK, Wong GTC, Lee EYP, Kulkarni D, Seow CS, Wong R, Co MT. ChatGPT versus human in generating medical graduate exam multiple choice questions: A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom). PLOS ONE, 2023.
ChatGPT wrote 50 questions with reference to Harrison's and Bailey & Love's, and two university teachers wrote 50 from the same two textbooks. Five independent international assessors scored all 100, numbered and shuffled, on five measures of quality. The total scores did not differ significantly. The ChatGPT questions scored lower on one measure, relevance (7.56 against 7.88), and their scores were spread more widely, while the human questions stayed within a narrower range. ChatGPT took about 20 minutes for its 50 questions and the two teachers about 211 minutes for theirs.
A systematic review of eight studies
Artsi Y, Sorin V, Konen E, Glicksberg BS, Nadkarni G, Klang E. Large language models for generating medical examinations: systematic review. BMC Medical Education, 2024.
The review searched MEDLINE for studies published up to November 2023 on multiple choice questions that large language models wrote for medical exams, and included eight. Five showed that the models can write competent questions valid for medical exams, and three did not evaluate whether the questions were valid. All eight presented faulty questions judged inappropriate for a medical exam, and some questions needed changes before they qualified. The authors conclude that, until more conclusive evidence is in, the models may serve as a supplementary tool for writing medical exams.
The studies had people judge each question, while the counts at the top of this page come from automatic checks, so the two sets of figures measure different things and cannot be compared one to one.
How to check an AI question yourself in twenty seconds
- Find the sentence. If the question does not show where it came from, search the document for its key phrase. If the phrase is not there, the question is not from your material.
- Cover the key and answer from the quote. Read only the quoted sentence and pick an option. If the quote does not settle it, the question is testing something the quote does not say.
- Look at the options. One option much longer than the rest, an option that repeats the stem, "all of the above": each is a tell, and each is a reason to distrust the question, not to learn from it.
What it means for exam preparation
A set of questions that survived these checks is a set you can practice on without re-reading the chapter to make sure each one is real. The rejected share is not wasted effort; it is the part of any AI-generated question bank that a student using an unchecked tool would have studied from. The free plan on this site runs these checks too.
Questions students ask
Does a high rejection rate mean a bad model?
Not on its own. A model that writes ambitious clinical vignettes loses more of them to the second model than one that writes recall questions, and a long run from a short chapter loses questions to the duplicate check because the chapter has only so many facts. The rate is a property of the model, the material and the run together, which is why the report is published with the number it was counted from.
Are the questions that pass guaranteed to be correct?
No. The checks catch a question whose quote is on neither the page it cites nor a page beside it, one a second model answers differently from the key, one that repeats an earlier question, and a set of writing and grounding flaws. They cannot judge the medicine. That is why every kept question is shown with its quote and page number, so the student can read the line the question came from.
Why does a repeated question count as a rejection?
Because a generator that is asked for forty questions from a twelve page handout will restate the same twelve facts in different words if nothing stops it. Each new question is compared with everything already written from the document, and a near repeat is thrown out rather than handed to the student as new practice.
No account needed to try it. Registering keeps your library, grades and review queue across devices.
Published , updated .