Skip to statewide header Skip to site header Skip to main content Skip to site footer Skip to statewide footer

2025-051 State High Risk: State Bar Exam

Audit Scope and Objectives

This audit is mandated by Business and Professions Code section 6145(d), which requires the State Auditor to conduct an audit of the February 2025 bar exam that evaluates the administration of the exam and how potential problems with the exam occurred. The State Auditor has developed the following objectives to guide the performance of this audit. Consistent with generally accepted government auditing standards, the State Auditor may refine or adjust these objectives as work is performed if he determines such changes are necessary.

1. Review and evaluate the laws, rules, and regulations significant to the audit objectives.

2. Evaluate the bidding and contracting process that the State Bar engaged in that led to awarding the contract to Proctor U, Inc., doing business as Meazure Learning, as well as the final terms of the contract, and determine the following:

  1. Whether the State Bar followed existing laws, regulations, and policies during the bidding and contracting process.
  2. What evaluation criteria the State Bar used to determine whether Meazure Learning had sufficient experience with conducting an examination similar to the State Bar examination.
  3. Whether the State Bar appropriately evaluated and authorized any changes, including cost changes, to the executed contract.
  4. Whether the State Bar had appropriate protections from, and evaluations of, any potential conflicts of interest that may have existed between relevant staff at the State Bar and Meazure Learning.

3. Evaluate the bidding and contracting process that the State Bar engaged in that led to awarding the contract to Kaplan for administration of the State Bar examination multiple choice questions, as well as the final terms of the contract for those services, and determine the following:

  1. Whether the State Bar followed existing laws, regulations, and policies during the bidding and contracting process.
  2. What evaluation criteria the State Bar used to determine whether Kaplan had sufficient experience with creating relevant and appropriate questions similar to existing comparable entities like the NCBE.
  3. Whether the terms of the contract allowed for oversight and monitoring of the question development process, and whether the State Bar appropriately utilized its oversight and monitoring to ensure that sufficient processes were engaged in to develop questions, ensure accuracy, and ensure fairness in the question development.
  4. Why Kaplan provided 100 multiple choice questions for the February 2025 bar exam, instead of 200.

4. Evaluate the process that the State Bar engaged in leading up to the administration of the February 2025 State Bar examination to ensure that the examination was conducted in a way that allowed participants to engage fairly in the examination process, and determine all of the following:

  1. When and how the State Bar became aware of any potential problems with the administration of the examination, including limits or problems with examination locations, and how those potential problems were addressed.
  2. How the process for remote examination employed for the February 2025 State Bar examination differed from the processes used for remote examination during the COVID-19 pandemic, and the reasons for those differences.
  3. The process and reasoning for determining that a makeup examination date should be offered, and the timeline for making that determination.
  4. The process for determining what specific equipment would be permitted for test-takers, such as white boards, and the reasoning and processes utilized to make any changes to those requirements leading up to the examination date.

5. Evaluate the events that led to the use of artificial intelligence to create multiple choice questions for use on the February 2025 State Bar examination, including all of the following:

  1. When the decision was made.
  2. Who or what department made the decision.
  3. The rationale for making the decision.
  4. When executive leadership at the State Bar knew that artificial intelligence was used to create questions for the February 2025 State Bar examination.
  5. What evaluation criteria were used to ensure that the questions were correct and appropriate for use on the February 2025 State Bar examination.
  6. What deficiencies in oversight by executive leadership at the Office of Admissions existed, if any, and how have any such deficiencies been remedied.

6. Itemize the State Bar costs arising from the administration of the February 2025 State Bar examination, including, but not limited to costs arising from existing contracts, legal representation, remedies provided to takers of the February 2025 State Bar examination, and any other associated costs. Itemized costs should include money already spent, money the State Bar is legally obligated to spend going forward, and other anticipated costs.

  1. Identify how much money the State Bar claimed it would save by using the February 2025 State Bar examination format and compare that amount to the total of the itemized costs.

7. Review and assess any other issues that are significant to the audit.

Opens in new window