Skip to main content

OSCE Evaluation Project

We commissioned a review of the Objective Structured Clinical Exam (OSCE). The project is now complete.

updated Feb. 26, 2014

posted May 23, 2012

CRNBC Nurse Practitioner OSCE Evaluation Project — January 2012

Project Description

In B.C., people who want to practise as full-scope nurse practitioners must pass the Nurse Practitioner Objective Structured Clinical Exam (OSCE, pronounced Os-key). There are other requirements as well, but the focus of this project was the OSCE. In April 2011, the College of Registered Nurses of B.C. commissioned a full review and evaluation of the OSCE.

The review included:

  • An analysis of development, scoring, and administration processes;
  • An outline of the exam’s strengths in measuring competencies; and 
  • Recommendations for improvement.
Background

To practise as a nurse practitioner (NP) in B.C., a person must be registered with the College. When registering, NPs choose one stream of practice, depending on their education: family, adult or pediatric. NPs registered elsewhere in Canada, may apply for similar registration in B.C.

The College recently changed its NP exam requirements to comply with the Agreement on Internal Trade and the Labour Mobility Act. In the fall of 2011, the College amended its bylaws and the B.C. government approved the changes. B.C. graduates and certain applicants (those with unrestricted registration as an NP elsewhere in Canada) no longer have to pass the OSCE to register with the College.

Also changed was the Scope of Practice for Nurse Practitioners: Standards, Limits and Conditions. New requirements were added for NPs performing restricted activities and the supervision of NP restricted activities.

Requirements for NPs performing restricted activities independently are in Section A: Standards, Limits and Conditions on Performing Restricted Activities. The process nurses follow when supervising an NP’s restricted activities is in Section B: Standards for Supervision of Nurse Practitioner Restricted Activities. These changes came into effect on October 1, 2011. The OSCE is no longer a registration requirement. Instead, it is a requirement for independently performing restricted activities. To identify who can, NPs who passed the ANCC exam and the OSCE are called “exam qualified.” NPs who are not “exam qualified” can only perform restricted activities under the supervision of an exam qualified NP or authorized physician.

Methodology/Approach

The project time frame spanned eight months (65 project days) from June, 2011 to January, 2012, during which quantitative and qualitative data was collected. There was a literature review, questionnaires, interviews, focus groups, stakeholder consultation and site visits.

The OSCE is a major tool for assessing clinical competence, as shown by the literature review. The review included examining the OSCE in measuring competencies, OSCE strengths and limitations, and key technical and organizational issues. The review also included best practices in OSCE design, scoring and administration, future trends and areas for development.

After studying the standards and best practices in the literature, a framework for review and evaluation of the OSCE was developed. (Vleuten, 2005, Boulet, 2008, Norcini et al., 2011). The framework describes the best practices needed for effective OSCE development, scoring and administration.

The first set of criteria looked at reliability and validity. The questions to be answered were: Can the result be repeated? What is the chance of error?

The second set of criteria looked at the organization and included acceptability, transparency and feasibility. The questions to be answered were: Do stakeholders accept the process? Is communication clear and appropriate? Is it practical, realistic, and sensible, given the circumstances and available resources?

The framework combines best practice with project design. These include data sources, methodology, whether criteria was met/not met and recommendations. The framework provides direction and helps the College interpret findings.

Findings

OSCE Strengths

  1. The OSCE met all technical needs for reliability and validity based on the best practices.
  2. The College has improved the OSCE since introducing it in 2005. In particular, improvements in the past two years considerably strengthened the exam from a technical perspective.
  3. The College has built a strong OSCE team. The team provides expertise and guidance. They are focused, committed, productive and adaptable.

OSCE Limitations

  1. The OSCE only partially met the organizational (as defined in the literature) needs for acceptability, transparency and feasibility based on the best practices.
  2. There is stakeholder support for the OSCE in setting a standard for the profession. However, there is much less support for the College in its role of exam developer and administrator.
  3. There are NPs and NP educators who are concerned about the College’s competence in OSCE development, scoring and administration.
  4. Many NPs did not offer their opinion in this review. Without their participation, we cannot know their views.
  5. The pass rates for the OSCE must improve.
Summary

The OSCE meets the technical requirements of reliability and validity. However, it only partially meets the organizational requirements of acceptability, transparency and feasibility. The College needs to work with stakeholders to resolve the limitations identified in the review. It also needs to provide more useful information about the exam.

Although several sources confirmed the reliability and validity of the exam, this view was not shared by all. There are negative views about design, scoring and administration (e.g., lack of exam reliability and validity, transparency, feasibility, etc.). These negative views detract from those of the majority, who endorse the exam. There is less support for the exam today than when it first arrived. Some negative views about the exam may spring from contextual and environmental factors.

Context and the environment

The OSCE developed at the same time as other changes occurred:

  • Introduction of a ‘fledgling nurse practitioner profession’ in B.C.;
  • The College’s role shifted from a professional association to a regulatory body; and
  • A regulatory framework developed that launched the OSCE as a requirement for NP registration.

Add to the mix:

  • Recent changes to registration requirements and scope of practice;
  • Stakeholder concerns with the exam;
  • Lack of clarity on the College’s role;
  • The still early stage of development for the profession and the exam; and
  • Provincial differences in NP education, scope of practice, regulatory approaches.

Some negative views are based on misinformation. These can be addressed through better communications and information sharing. Other problems need a focused and collaborative effort to lay the foundation for the future success of the OSCE.

6 Recommendations
  1. Return the exam to being a registration requirement for nurse practitioners in B.C.
  2. Listen to and engage stakeholders. Work with key stakeholders to address the recommendations identified in the OSCE evaluation project.
  3. Put in place strategies to improve understanding and support of the OSCE. Build trust through better transparency and access to useful information.
  4. Work with stakeholders to address low pass rates. A realistic goal could be to work towards similar pass rates of comparable exams.
  5. Consider finding a third party to take over as exam developer. Keep the exam developer separate from the College and universities. 
  6. Share the evaluation project report with stakeholders.

The College’s Response to the Report

OSCE & Registration Requirement

  1. We are working with government to eliminate the two-tiered process.
  2. We are working with other Canadian nurse regulators on a national project which may lead to a national approach to NP exams.

Collaborative Working Group

We will explore the idea with the B.C. Nurse Practitioner Association and NP educators in the coming months.

Marketing Communication Plan

We will assess our communications and make improvements as needed.

Pass Rates

  1. Pass rates have improved since June 2011. We will continue to monitor exam results. We have improved communications about exam performance/pass rates and preparation needs. We continue to solicit feedback to ensure improvements are ongoing.
  2. We are working on the latest exam report for educators. Each NP program receives a report showing how their students as a whole performed on the exam.

Move Exam

We will consider this recommendation as we work with other stakeholders on the national project.

Share Report

Implemented

 OSCE Evaluation FAQs

Why did the College decide to evaluate the NP OSCE?
The College had improved the OSCE process during the past six years based on feedback from candidates and other stakeholders. However, the College recognized the value of a comprehensive and external review of the exam process. We expect the external evaluation to serve as an important evidence-based framework for improvements to the exam process in assessing competencies required for nurse practitioner practice in B.C.
What was the tendering process for the OSCE evaluation?
In April 2011, the College put out a Request for Proposal (RFP) to identify a qualified consultant to do an evaluative review of the development, scoring and administration of the OSCE. The RFP was posted electronically on the B.C. Bid and Merx websites. A consultant who met the criteria for conducting the evaluation was selected from a number of candidates.
How were nurse practitioners invited to participate in the review?
An invitation to participate in a focus group was sent to all NPs who had taken the exam in 2009 and 2010 (about 60).  The College welcomed feedback from a range of NPs, including those who passed the exam the first time and those who took it more than once. NPs who were not able to attend the focus group, but wished to participate could complete a questionnaire. Unsuccessful OSCE candidates who did not continue on and were not registered were also contacted to participate in phone interviews.
What criteria were used in selecting the consultant?
  • Be knowledgeable of competence assessment.
  • Have experience in research methods and data analysis and synthesis of information as it pertains to the subject area.
  • Have proven ability in doing projects at health system and national levels.
  • Have excellent written and verbal communication skills.
  • Understand and comply with the confidentiality requirements of the Health Professions Act and Freedom of Information and Protection of Privacy Act. This includes not discussing, producing, reproducing or publishing any examination material or content except as authorized by the College.
  • Hold a masters degree in a health-care field.
  • Be unbiased without professional conflict of interest.
What components of the OSCE process were evaluated?

Three components were evaluated:

  1. Development of OSCE, e.g., process for writing, revising, selecting, and implementing case scenarios, recruitment of examiners, process for validating exam content.
  2. Scoring of OSCE, e.g., process for standard setting (setting pass score), process for marking exam, process for addressing reliability of marking and scoring.
  3. Administrative processes, e.g., selection of venue, organization of exam materials, orientation of candidates.
What stakeholders were included in this review?
The consultation included several stakeholder groups comprising nurse practitioners, faculty from B.C. nurse practitioner programs, College staff, consultants used by the College, and others involved in the College’s OSCE or OSCEs administered for other health-care professions.
Why was the timeframe of the evaluation project extended?

The consultation period took longer than anticipated. There was also a larger than expected amount of qualitative data to analyze.

 Learn more

Home > About CRNBC > Announcements > 2012 Announcements > OSCE Evaluation Project