Rejection rates by model family
Every question written for this site goes through the same checks, whichever model wrote it. This is how many of them each model family loses, over the last 30 days, with the number the rates are counted from.
These rates are counted from the call log, which has been kept since September 14, 2026. The running totals printed elsewhere on the site count every question since the site opened on September 11, 2026, so the two sets of numbers cover different periods and are not meant to add up to each other.
Counted over to : 2,403 generation calls that returned 6,071 questions. A family is published once it has returned 200 questions in the window; below that it says so rather than printing a rate from a handful of calls.
| Model family | Questions written | Quote not found on the cited page | Second model disagreed | Duplicate | Flaw | Not asked for or too late | Accepted |
|---|---|---|---|---|---|---|---|
| qwen | 4,545 | 6.1% | 6.1% | 13.9% | 28.8% | 2.1% | 43.0% |
| deepseek | 437 | 10.3% | 2.3% | 2.7% | 11.9% | 2.3% | 70.5% |
| glm | 410 | 2.0% | 7.3% | 5.6% | 51.5% | 2.2% | 31.5% |
| nemotron | 393 | 3.1% | 0.5% | 2.5% | 53.9% | 1.3% | 38.7% |
| gpt-oss | 273 | 7.7% | 1.5% | 1.8% | 39.2% | 1.8% | 48.0% |
| openrouter-primary | 13 | not enough data | not enough data | not enough data | not enough data | not enough data | not enough data |
| gemma | 0 | not enough data | not enough data | not enough data | not enough data | not enough data | not enough data |
By lane
The free lane runs DeepSeek V4 Flash on Cloudflare Workers AI first, with other open-weight models as fallbacks. The paid lane runs the priority models: DeepSeek V4 Pro writes and Kimi K2.6 answers blind. Both lanes are held to the same checks and both are counted here, which is why this page is about model families rather than about lanes.
| Lane | Questions written | Quote not found on the cited page | Second model disagreed | Duplicate | Flaw | Not asked for or too late | Accepted |
|---|---|---|---|---|---|---|---|
| free | 6,028 | 6.0% | 5.3% | 11.3% | 31.3% | 2.1% | 44.0% |
| paid | 43 | not enough data | not enough data | not enough data | not enough data | not enough data | not enough data |
What each column means
- Questions written: questions the model returned in a parseable answer. Answers that could not be read as JSON are counted separately in the CSV, per call, because they returned no questions to count.
- Quote not found on the cited page: the verbatim quote was not in the stored text of the page the model named, or it cited a page outside the section it was given. The CSV splits those two.
- Second model disagreed: a model from a different family, shown the question and its quotes but never the answer key, picked a different answer.
- Duplicate: the question repeated one already accepted from the same document.
- Flaw: a structural or quality rule rejected it, such as an option set with a giveaway or a lead-in with no single best answer. Those rules are the ones the MCQ flaw checker runs in your browser.
- Not asked for or too late: the question was sound but never reached a student anyway. Either it was a type the job did not ask for (a short answer item in a multiple choice job), or the job filled up from other calls while this one was being written. Both are counted so the row adds up; neither is a mark against the model.
- Accepted: what survived all of it and reached a student.
Method
Every number here comes from production jobs on this site. One row is written per provider call, at the moment the call's outcome is known, holding counts and nothing else: no question text, no document text and no user. The counts are attributed to the model that actually answered, which is not always the model the job asked for, because the pool falls back to another model when one is rate limited or slow. Attributing a fallback to the model that did not run would publish a number about the wrong model.
The window is the last 30 days, recounted at most twice an hour, and it is one of three fixed windows. Rates are shares of the questions a family returned, and every question a call returned is in exactly one of these columns, so they add up with the accepted share to the whole. A call that came back to a job which had already finished writes no row at all: its questions were never looked at, and a model that answered late is not a model that answered badly. A family under 200 returned questions is listed as not enough data, because at that size one bad section moves a rate by several points.
Download the same numbers as CSV, or read them as JSON. The numbers are published under the Creative Commons Attribution 4.0 license: use them, quote them, and name MedUni Exam as the source.
Updated September 24, 2026.
Generate from your own chapter
No account needed to try it. Registering keeps your library, grades and review queue across devices.
Published , updated .