Report 2011-117 Data Reliability Assessments

Education, Department of
Description of Data Agency Purpose of Data
California High School Exit Examination results

To provide for the development, administration, scoring, analysis, and reporting of the California High School Exit Examination (exit examination).

Magnitude of Data

The exit examination data for the 2007-08 school year contains more than one million student test records.

Purpose of Testing Data Reliability Determination

To identify students' performance on their first attempt at the exit examination during the second half of grade 10.

Undetermined Reliability—We performed data-set verification procedures and electronic testing of key data elements and did not identify any issues.

The department contracted with Educational Testing Service (ETS) to provide for the development, administration, scoring, analysis, and reporting of the exit examination. ETS subcontracted for the design, printing, and scanning of hard-copy student answer documents.

We were unable to perform accuracy and completeness testing as required by government auditing standards because ETS' subcontractor previously destroyed the hard-copy student answer documents in accordance with the terms of its contract.

Alternatively, we interviewed officials at the California Department of Education (department) and reviewed relevant documentation to determine whether it conducted any work to verify that student essays were accurately scored. We learned that the department contracted with Human Resources Research Organization (HumRRO) to conduct an independent evaluation of the exit examination test results through September 2008. As part of its evaluation, HumRRO analyzed the consistency with which student essays were scored for the 2007-08 school year and found that the results were generally comparable to the 2006-07 school year and somewhat improved in comparison to previous years.

We also interviewed officials at the department, ETS, and ETS' subcontractor and reviewed relevant documentation to determine whether they conducted any work to verify the accuracy of the electronic file of student multiple-choice answers against the hard-copy answer documents. Although ETS asserted to us that it performed quality control measures related to the accuracy of the data, it was not able to produce the detailed results of this analysis. We found that the level of detail ETS provides to the department when reporting the results of its quality control measures could be improved. This was communicated to the department in a separate management letter.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
California Longitudinal Pupil Achievement Data System (CALPADS)

CALPADS was created to enable California to meet federal requirements delineated in the No Child Left Behind (NCLB) Act of 2001, which increases accountability for student achievement.

Magnitude of Data

As of August 15, 2011, CALPADS contained more than 23 million records of student enrollment.

Purpose of Testing Data Reliability Determination

To calculate dropout and graduation rates among various demographic categories and by California High School Exit Examination (exit examination) result for the 2009-10 cohort.

Undetermined Reliability—We performed data-set verification procedures and electronic testing of key data elements and did not identify any issues.

We did not perform accuracy and completeness testing because the source documentation for this system is not centrally stored at the California Department of Education, thereby making such testing impractical.

Alternatively, we examined 20 student records from each of the six school districts we visited to determine how schools assigned exit codes and whether they conformed to the department's instructions.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A

Report type

Report type
















© 2013, California State Auditor | Privacy Policy | Conditions of Use | Download Adobe PDF Reader