Item Analysis: How to Evaluate and Improve Certification Exam Questions

What Is Item Analysis?

Item analysis is the process of evaluating individual exam questions using candidate response data. It helps testing administrators identify questions that are too easy, too difficult, ambiguous, or ineffective at measuring the intended knowledge.

For professional certification programs, item analysis provides evidence about how an examination performs after candidates begin taking it. That evidence helps you decide which questions to retain, revise, or retire.

It also supports a more defensible exam development process. When your certification board asks why a question was changed, you should be able to explain the decision using documented review criteria rather than intuition alone.

Three measurements form the foundation of most traditional item analysis processes:

• Item difficulty: The proportion of candidates who answer a question correctly.
• Item discrimination: How well a question distinguishes between higher- and lower-performing candidates.
• Distractor analysis: How candidates respond to incorrect multiple-choice answer options.

Together, these measurements help your team evaluate question quality, identify potential scoring problems, and improve future exam administrations.

---

The Three Core Metrics in Item Analysis

1. Item Difficulty (P-Value)

Item difficulty measures the percentage of candidates who answer a particular question correctly. Despite its name, a higher item difficulty value indicates an easier question.

The calculation is straightforward:

For example, if 100 candidates receive a question and 75 answer correctly, the item’s p-value is 0.75.

Question	Correct Responses	P-Value	Interpretation
A	95 of 100	0.95	Very easy
B	65 of 100	0.65	Moderate
C	20 of 100	0.20	Difficult

These are illustrative results, not universal classification thresholds. The appropriate difficulty depends on the purpose of the exam and the competency being measured.

A safety certification might intentionally include a fundamental question that 95% of qualified candidates answer correctly. That does not make the question unnecessary. It may measure knowledge that every certified professional must demonstrate.

Conversely, a question with a 0.20 p-value warrants investigation, but its difficulty alone does not establish that it is defective.

2. Item Discrimination

Item discrimination measures how strongly performance on an individual question relates to overall examination performance.

A useful question generally produces higher correct-response rates among candidates who demonstrate stronger knowledge of the tested material.

One approach compares the percentage of correct responses between high-performing and low-performing groups.

Consider a hypothetical certification exam:

• 85% of high-performing candidates answer Question 12 correctly.
• 35% of low-performing candidates answer the same question correctly.
• The upper-minus-lower discrimination index is 0.50.

This pattern suggests the question is distinguishing between the two groups.

Now consider Question 18, where 40% of high-performing candidates answer correctly, compared with 65% of low-performing candidates.

That produces a negative discrimination index of -0.25.

Negative discrimination deserves investigation. It may indicate an incorrect answer key, ambiguous wording, or a question that measures something different from the broader assessment.

More advanced analyses use statistics such as the point-biserial correlation, which measures the relationship between an item’s score and overall test performance. This is not the same calculation as the upper-minus-lower discrimination index.

Neither measure should be interpreted without considering sample size, exam structure, and the content being tested.

3. Distractor Analysis

Distractor analysis evaluates the incorrect answer choices in multiple-choice questions.

Effective distractors represent plausible errors or misconceptions. They should attract some candidates who have not mastered the underlying concept without misleading candidates who understand it.

Consider the following hypothetical response distribution:

Answer	Selected By	Review Consideration
A (Correct)	58%	Correct response
B	35%	Check for ambiguity or misconception
C	6%	Review effectiveness
D	1%	Possibly ineffective distractor

Option B deserves attention. Its popularity may reveal a common misconception, but reviewers should also determine whether its wording creates a defensible alternative answer.

Option D might be too obviously incorrect. However, response frequency alone cannot establish distractor quality. You also need to consider which candidates selected each option.

The goal is not to distribute responses evenly. It is to ensure that each answer choice contributes to the intended assessment.

---

How to Conduct Item Analysis for a Certification Exam

For a professional certification program, a practical item analysis process should connect statistical findings to documented operational decisions.

This sequence matters. An administrator should not automatically delete every item with a low discrimination statistic. The result is a review signal, not a final judgment.

---

What Should an Item Analysis Report Include?

A useful item analysis report should support decisions by certification administrators, subject-matter experts, and psychometric reviewers.

At minimum, consider including these fields:

• Item identifier: A stable reference that survives question-bank updates.
• Exam and form: The assessment and version where the item appeared.
• Blueprint domain: The competency or content area being assessed.
• Response count: The number of valid candidate responses.
• Item difficulty: The proportion of correct responses.
• Item discrimination: An appropriate discrimination statistic.
• Answer distribution: Selection frequency for each response option.
• Review status: Retained, flagged, revised, or retired.
• Decision history: Reviewer, date, rationale, and approval.

These fields connect question performance with the administrative work needed to maintain an examination.

Without that connection, your team may end up maintaining separate spreadsheets, reviewer notes, and item-bank records. That increases the effort required to reconstruct why a question changed.

For larger programs, an effective reporting process may also require reliable item-version identifiers, exposure tracking, and comparisons across administrations.

---

Item Analysis, Test Blueprints, and Exam Reliability

Item analysis is one part of a broader examination quality program. It should not operate independently of exam design, content coverage, scoring, or reliability evaluation.

Connect Findings to the Exam Blueprint

Your exam blueprint identifies the competencies and content domains that the assessment must cover. Item statistics help you evaluate how questions within those domains perform.

For example, a trade association might discover that questions in one competency domain consistently produce lower correct-response rates than expected.

That finding does not automatically mean the questions should become easier. The committee should investigate whether the content is appropriately challenging, whether the blueprint remains accurate, and whether the candidate population has changed.

Test Blueprints in Gauge Online

Understand the Limits of Item Statistics

Item difficulty and discrimination do not independently establish examination validity. They describe how questions function within a particular dataset.

Other evidence may be needed to evaluate score reliability, content relevance, passing-score decisions, fairness, and consistency across exam forms.

For programs using randomized question selection, analysis also requires attention to which candidates actually received each item. Comparing the same question across different candidate groups can produce misleading conclusions if those groups differ substantially.

The National Board of Medical Examiners’ item-writing guidance explains how item difficulty, discrimination, and response-option analysis contribute to evaluating assessment questions.

---

How Item Analysis Supports Certification Accreditation

Certification programs pursuing NCCA accreditation or evaluating ISO/IEC 17024 requirements must pay attention to how they develop, administer, and maintain their assessments.

Item analysis can provide useful supporting evidence within these processes.

A documented review process may demonstrate that your organization:

• Monitors examination question performance.
• Investigates unexpected or potentially problematic results.
• Uses qualified reviewers to evaluate question quality.
• Records decisions about question revisions or retirement.
• Maintains evidence of ongoing examination quality assurance.

These records can support a broader accreditation evidence package, but item analysis alone cannot establish compliance with an accreditation standard.

Your program also needs appropriate documentation of examination development, governance, content validity, scoring practices, security, and other applicable requirements.

The operational advantage of a structured review process is straightforward: when an auditor requests evidence, your team can locate the relevant statistics and decisions without reconstructing the history manually.

NCCA Accreditation and ISO 17024 Standards

---

Common Item Analysis Mistakes to Avoid

Treating Statistical Thresholds as Automatic Rules

Some item analysis guides provide suggested difficulty or discrimination thresholds. These may serve as screening criteria, but they are not universal standards.

An easy item may measure essential knowledge. A difficult item may appropriately assess advanced competency.

Review the item’s purpose before making changes.

Ignoring Small Sample Sizes

An item answered by 15 candidates offers less stable statistical evidence than the same item answered by 500 candidates.

For low-volume certification programs, it may be appropriate to combine comparable administrations, extend the review window, or involve a psychometrician in interpreting uncertain results.

Changing Questions Without Preserving History

Replacing question text without preserving the previous version can make historical performance difficult to interpret.

Maintain a record of what changed, when it changed, and why the change was approved.

Reviewing Statistics Without Taking Action

Generating an item analysis report does not improve an examination by itself.

Your program needs a repeatable process for assigning reviews, documenting decisions, updating question banks, and checking whether revisions improved performance.

---

Frequently Asked Questions About Item Analysis

What is a good item difficulty score for a certification exam?

There is no universal ideal. The appropriate difficulty depends on the competency being evaluated and the examination’s purpose. A fundamental safety question may appropriately have a very high correct-response rate.

---

What does negative item discrimination mean?

Negative discrimination indicates that lower-performing candidates answered a question correctly more often than higher-performing candidates, according to the measure used. It warrants review of the answer key, wording, content alignment, and underlying data before making changes.

---

How many candidate responses do you need for item analysis?

There is no single minimum that applies to every analysis. Larger samples generally produce more stable statistics, while smaller samples require additional caution. A psychometrician can help determine appropriate methods for your exam volume and stakes.

---

How often should certification programs perform item analysis?

Review item performance periodically as sufficient data becomes available. High-stakes programs may also conduct reviews after pilot administrations, major exam revisions, or unusual scoring patterns. The schedule should follow the program’s documented assessment quality procedures.

---

Is item analysis required for NCCA accreditation?

Item performance analysis can contribute evidence to examination quality and maintenance practices, but requirements must be evaluated against the applicable standards. Programs should document the methods they use and obtain qualified psychometric guidance when appropriate. A report alone does not establish accreditation compliance.

---

Turning Item Analysis Into a Repeatable Quality Process

Start by defining who reviews item statistics, how questions are flagged, and what evidence reviewers need to make decisions. Document each approved change and establish when revised items will be evaluated again.

If your certification program manages question banks, exam delivery, and credentialing through disconnected systems, these reviews become harder to coordinate. Gauge Online helps organizations manage online examinations and certification workflows within a unified platform. When evaluating a platform for a formal item analysis process, confirm that the available response data and reporting meet your psychometric requirements.