How We Keep Our PMP Questions Accurate (and Catch the Bad Ones)

Quick answer: Every ExamElephant question is drafted with AI assistance and reviewed by PMP-credentialed practitioners before it can be served. After publication, discrimination statistics and top-quartile distractor analysis flag likely miskeys and ambiguous items, and every learner flag is read by a human — nothing auto-closes. Items found wrong are corrected or retired, re-approved before serving again, and your mastery state on them is reset.

A miskeyed practice question hurts you twice: once when it marks your correct reasoning wrong, and again when you dutifully memorize the wrong answer. Ambiguous questions are nearly as bad — they teach you that the material is arbitrary, which corrodes the trust that makes practice useful at all.

Every question bank contains mistakes, ours included. The difference between banks isn't the existence of errors — it's how fast errors get caught. Here is our full quality pipeline, including the part vendors usually leave out.

Before publish: draft → review → approved

We are straightforward about this: our items are drafted with AI assistance, and no item reaches learners without human review. Every question lives in a draft → approved workflow, and only approved items can ever be served.

Human reviewers — practitioners who hold the PMP credential — check each draft for:

• Key correctness. Is the credited answer defensibly the single best answer, not merely a correct one?
• Distractor quality. Wrong options must be plausible to the underprepared and clearly inferior to the prepared — attractive-but-wrong, never trick-wrong.
• Rationale completeness. The explanation must say why the key is right and why each distractor fails. If a reviewer can't articulate why an option is wrong, the item doesn't ship.
• Blueprint fit and style. Alignment to the 2026 ECO (live since July 9, 2026), one testable idea per item, no "all of the above," no gotcha negatives.

Review catches most problems. Not all of them. Some flaws only surface once hundreds of real learners answer an item — which is where the statistics take over.

After publish: statistics that catch what reviewers miss

Item discrimination

For every question we track a discrimination statistic — a point-biserial correlation. In plain terms: do learners who score well overall get this item right more often than learners who score poorly? A good item discriminates positively. An item near zero tells us nothing about ability. A negative discriminator — strong learners missing it more often than weak ones — is the statistical smell of a miskey or a broken stem, and it triggers automatic re-review.

Top-quartile distractor analysis

For flagged items, we look at what the top quartile of scorers actually chose. If the strongest learners converge on one specific "wrong" option, that is the classic miskey signature: the best-prepared people in the bank are collectively telling us the key is wrong, or that two options are genuinely defensible. Those items leave circulation until they're re-adjudicated.

This distinction matters because hard and broken look identical in raw accuracy. TCPI items run under 40% first-attempt accuracy in our pilot-bank data — but they discriminate positively, so they're hard, not broken. A 40% item that the top quartile also fails is a different animal entirely.

Difficulty outliers

Items that land far from their expected difficulty get a human re-read for ambiguity even when discrimination looks acceptable. Sometimes the fix is one clarifying word in the stem. The fixed item still goes back through the approval workflow before it serves again.

Learner flags close the loop

Every question has a flag control, and every flag gets human eyes — no auto-close. Most flags turn out to be the productive kind of disagreement a rationale can resolve; a minority are real catches. We would rather re-review ten defensible items than defend one bad one.

The stakes are higher for us than for a static question bank, because our retest engine brings your misses back at 3 and 7 days. A bad item wouldn't appear once — it would keep coming back until you "mastered" something false. In a retest product, quality control isn't a nice-to-have. It's load-bearing.

What happens when we're wrong

When re-review concludes that a key or a stem was wrong, we don't quietly swap it:

• The item is corrected — or retired — and passes through draft → approved again before it can serve.
• The rationale is rewritten to address the old key directly, because some learners will have memorized it.
• Your mastery state on that item is reset fairly, so you aren't penalized for having "missed" a miskeyed question.

No secret sauce, just discipline

Nothing above is proprietary genius. Discrimination statistics and distractor analysis are standard psychometrics, decades old. What varies across the prep industry is whether anyone actually runs them, and whether a human adjudicates the results. We publish this so you can hold us to it.

Judge the output yourself: signing up is free and includes 75 questions with full rationales — every one of them a survivor of the pipeline described here.

FAQ