Report 2012-107 Data Reliability Assessments

Controller's Office, State
Description of Data Agency Purpose of Data
California Department of Developmental Services' data as maintained in the California State Controller's Office's (state controller) Uniform State Payroll System (payroll system)

To process the State's payroll and personnel transaction documents.

Magnitude of Data

Payroll data maintained by the state controller's payroll system for 2008 through 2012 for the following departments: California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services.

The payroll system contained more than 36.6 million payroll and personnel transactions for the period 2008 through 2012.

Purpose of Testing Data Reliability Determination

To calculate the total amount of regular and overtime pay, the average number of employees by pay type for each of the departments listed (California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services), and overtime pay as a percent of total earnings--the sum of regular and overtime pay-- for 2012.

To calculate the total amount of regular and overtime pay, and overtime pay as a percent of total earnings, by employee class title for each of the departments listed for 2012.

To calculate overtime paid as a percent of total earnings for the California Department of Developmental Services' (Developmental Services) Office of Protective Services' (OPS) employees as compared to all other department employees by location and department-wide for each year between 2008 and 2012.

For a selection of 62 Developmental Services' employees who were paid the same amount in overtime pay as regular pay for the period of 2008 through 2012, calculate the total amount of regular pay and overtime pay, and the number of employees by class title.

For a selection of eight Developmental Services' employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012, identify the employee's class title and work location, calculate the average weekly hours worked by calendar year, and total the employee's regular pay and overtime pay.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We relied on the completeness testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2012. Since the State's financial audit for fiscal year 2012-13 is still in progress, we cannot rely on this report to verify completeness between July 2012 and December 2012. However, because we found the payroll data to be complete between January 2008 and June 2012, we have reasonable assurance that the payroll data for the period of July 2012 through December 2012 are also complete. Consequently, we found the data to be complete.

We relied on the accuracy testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2011. For the period July 2011 through December 2012, we performed accuracy testing on a selection of 10 regular or overtime payroll transactions by tracing key data elements to supporting documentation and found no errors.

We performed additional accuracy testing on two key data elements—overtime hours paid and work location—for the selection of 62 Developmental Services employees by tracing key data elements to supporting documentation and found no errors.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
California Department of Developmental Services' data as maintained in the California State Controller's Office's California Leave Accounting System (leave accounting system)

To perform a variety of functions necessary to accurately track leave system eligibility, state service credits, and leave benefit activity.

Magnitude of Data

The leave accounting system contained more than 36.7 million leave transactions for the period 2008 through 2012.

Purpose of Testing Data Reliability Determination

To calculate the total accrued compensating time off leave hours for a selection of eight California Department of Developmental Services (Developmental Services) employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We performed accuracy testing on a random sample of 31 total unique payroll transactions—selected from 62 Developmental Services employees who were paid at least the same amount in overtime as regular pay for the period of 2008 through 2012. Since our initial random sample of 29 only contained three transactions where the employee accrued compensating time off, we augmented our testing by selecting the next two transactions in our sample where the employee accrued compensating time off hours to gain additional assurance of the accuracy of these hours. We found no errors.

To verify completeness, we used the initial random sample selected for accuracy testing—described in detail above—in which we selected 29 unique payroll transactions. We traced payroll timekeeping documents to the leave accounting data to determine if accrued compensating time off was properly recorded in the leave accounting data. We found no errors.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Corrections and Rehabilitation, Department of
Description of Data Agency Purpose of Data
California Department of Corrections and Rehabilitation's data as maintained in the California State Controller's Office's (state controller) Uniform State Payroll System (payroll system)

To process the State's payroll and personnel transaction documents.

Magnitude of Data

Payroll data maintained by the California State Controller's Office's Uniform State Payroll System (payroll system) for 2008 through 2012 for the following departments: California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services.

The payroll system contained more than 36.6 million payroll and personnel transactions for the period 2008 through 2012.

Purpose of Testing Data Reliability Determination

To calculate the total amount of regular and overtime pay, the average number of employees by pay type for each of the departments listed (California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services), and overtime pay as a percent of total earnings--the sum of regular and overtime pay-- for 2012.

To calculate the total amount of regular and overtime pay, and overtime pay as a percent of total earnings, by employee class title for each of the departments listed for 2012.

To calculate overtime paid as a percent of total earnings for the California Department of Developmental Services' (Developmental Services) Office of Protective Services' (OPS) employees as compared to all other department employees by location and department-wide for each year between 2008 and 2012.

For a selection of 62 Developmental Services' employees who were paid the same amount in overtime pay as regular pay for the period of 2008 through 2012, calculate the total amount of regular pay and overtime pay, and the number of employees by class title.

For a selection of eight Developmental Services' employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012, identify the employee's class title and work location, calculate the average weekly hours worked by calendar year, and total the employee's regular pay and overtime pay.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We relied on the completeness testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2012. Since the State's financial audit for fiscal year 2012-13 is still in progress, we cannot rely on this report to verify completeness between July 2012 and December 2012. However, because we found the payroll data to be complete between January 2008 and June 2012, we have reasonable assurance that the payroll data for the period of July 2012 through December 2012 are also complete. Consequently, we found the data to be complete.

We relied on the accuracy testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2011. For the period July 2011 through December 2012, we performed accuracy testing on a selection of 10 regular or overtime payroll transactions by tracing key data elements to supporting documentation and found no errors.

We performed additional accuracy testing on two key data elements—overtime hours paid and work location—for the selection of 62 Developmental Services employees by tracing key data elements to supporting documentation and found no errors.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Developmental Services, Department of
Description of Data Agency Purpose of Data
California Department of Developmental Services' Developmental Centers Division's Microsoft Excel files

To document and track incidents, such as resident abuse.

Magnitude of Data

The Microsoft Excel files contain information pertaining to more than 5,100 allegations of resident abuse and more than 80 cases sent to the district attorney for five developmental centers.

Purpose of Testing Data Reliability Determination

For Canyon Springs and Lanterman, determine the number of documented alleged resident abuse cases and of those cases the number investigated and substantiated.

For all five developmental centers, determine the disposition of alleged resident abuse cases, such as the number sent to the district attorney and the number that resulted in criminal complaints filed by the district attorney within the past five years.

Undetermined Reliability—We did not test the reliability of these data because it would not be cost-effective to trace this summary-level data back to the individual transactions that support the totals.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
Fairview Developmental Center's Incident Reporting Information System

To document and track incidents, such as resident abuse.

Magnitude of Data

The data contain nearly 530 records pertaining to incidents at Fairview Developmental Center from August 6, 2012 through December 31, 2012

Purpose of Testing Data Reliability Determination

Determine the number of documented alleged resident abuse cases.

Determine the disposition of those cases, such as the number investigated and the number substantiated.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified that key data elements matched source documentation. We found no errors.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hard copy files back to the Incident Reporting Information System. We found the data to be complete.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
Fairview Developmental Center's Incident Reporting Management Application

To document and track incidents, such as resident abuse.

Magnitude of Data

The data contain more than 11,800 records regarding incidents at Fairview Developmental Center from January 2008 through August 2012.

Purpose of Testing Data Reliability Determination

Determine the number of documented alleged resident abuse cases.

Determine the disposition of those cases, such as the number investigated and the number substantiated

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no significant issues.

To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified that key data elements matched source documentation. We found one error within the special investigation number data field. Therefore, we randomly selected another 17 incidents to verify the special investigation number data field matched source documentation. We found no additional errors.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hard copy files back to the electronic database. We found the data to be complete.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
Porterville Developmental Center's Risk Management Database

To document and track incidents, such as resident abuse.

Magnitude of Data

The data contain more than 5,700 records pertaining to incidents at Porterville Developmental Center from January 2008 through December 2012.

Purpose of Testing Data Reliability Determination

Determine the number of documented alleged resident abuse cases.

Determine the disposition of those cases, such as the number investigated and the number substantiated.

Not Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no significant issues.

Although accuracy testing is not required because we found the data to be incomplete, we decided the results of the testing would be important and informative. To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified that key data elements matched source documentation. We found no errors.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hard copy files back to the electronic database. We found one incident report that was not documented in the Risk Management Database. As a result, the data is incomplete.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
Sonoma Developmental Center's Incident Reporting Information System

To document and track incidents, such as resident abuse.

Magnitude of Data

The data contain nearly 225 records pertaining to incidents at Sonoma Developmental Center from November 2012 through December 2012

Purpose of Testing Data Reliability Determination

Determine the number of documented alleged resident abuse cases.

Determine the disposition of those cases, such as the number investigated.

Undetermined Reliability—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We did not perform accuracy or completeness testing because hard copy source documentation was not available for this review. Alternatively, following the U.S. Government Accountability Office's guidelines, we could have reviewed the adequacy of selected system controls that include general and application controls. However, we did not conduct these reviews because the data we are analyzing was for one location for a two month period. We determined that the level of effort required to perform these reviews was not cost effective.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
Sonoma Developmental Center's New Incident Reporting Management Application

To document and track incidents, such as resident abuse.

Magnitude of Data

The data contain more than 12,000 records pertaining to incidents at Sonoma Developmental Center from January 2008 through October 2012

Purpose of Testing Data Reliability Determination

Determine the number of documented alleged resident abuse cases.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no significant issues.

To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified that key data elements matched source documentation. We found no errors.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hard copy files back to the New Incident Reporting Management Application. We found the data to be complete.

Determine the number of documented alleged resident abuse cases that were investigated.

Not Sufficiently Reliable—We performed data-set verification procedures and electronic testing of this key data element and found no significant issues.

We did not test the accuracy of the number of resident abuse cases investigated because developmental center staff informed us that its employees do not consistently use the special investigation number—the data field that indicates whether a case was investigated—in the New Incident Reporting Management Application (NIRMA).

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hardcopy files back to NIRMA. We found the data to be complete.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
Sonoma Developmental Center's Office of Protective Services' Microsoft Excel files

To document and track incidents, such as resident abuse.

Magnitude of Data

The data contain more than 2,700 records pertaining to cases at Sonoma Developmental Center from January 2008 through December 2012, including data on 44 substantiated cases of abuse.

Purpose of Testing Data Reliability Determination

Determine the disposition of alleged resident abuse cases, such as the number substantiated.

Not Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no significant issues.

To test the accuracy of the developmental center's data, we randomly selected a sample of 29 incidents and verified whether the key data elements matched source documentation. We found two errors in the Office of Protective Services case number data field, seven errors in the case type data field, and six errors in the disposition data field.

To test the completeness of the developmental center's data, we haphazardly selected 29 incidents and traced them from hardcopy files back to the Microsoft Excel files. We found the data to be complete.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Public Health, Department of
Description of Data Agency Purpose of Data
Automated Survey Processing Environment (ASPEN)

ASPEN allows states and the Centers for Medicare and Medicaid Services to input and access both survey results and enforcement information.

Magnitude of Data

The data contain more than 360 licensing and certification survey visits conducted at five state developmental centers from July 2005 through June 2012, and more than 10,700 entity-reported incidents and complaints for those centers from July 2008 through June 2012.

Purpose of Testing Data Reliability Determination

To determine the classification and the amount of time it took the California Department of Public Health (Public Health) to initiate investigations into complaints and incidents.

Not Sufficiently Reliable—As we reported in a previous audit report, Department of Health Services: Its Licensing and Certification Division Is Struggling to Meet State and Federal Oversight Requirements of Skilled Nursing Facilities (2006-106, April 2007), the federal database Public Health uses to track its investigations, ASPEN, has weak controls that preclude Public Health (previously part of the Department of Health Services) from preventing erroneous data entry or detecting data errors for key data fields such as dates when complaints are received and investigated.

There were 10,746 complaints and incidents in our audit period and nine of them were assigned to more than one priority level. We did not include these nine in our testing. However, through our testing we discovered data limitations. Specifically, we found 863 of the 10,737 complaints and incidents we tested were either missing a date or had an investigation initiation date that occurred before the intake date. Although Public Health provided documents indicating that its investigation was initiated after the complaint or incident was received for four of these occurrences, we could not easily gather the documents needed to find the missing dates, evaluate illogical data, or rule out data entry errors for the remaining 859 records.

Public Health indicated that it has requested the federal government to make improvements to ASPEN.

To determine whether state licensing and federal certification surveys and certification revisits occurred when required.

Undetermined Reliability—Because the Department of Public Health indicated that it has requested the federal government to make improvements to ASPEN, we did not perform any further testing of the data we obtained from the federal database.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Social Services, Department of
Description of Data Agency Purpose of Data
California Department of Social Services' data as maintained in the California State Controller's Office's (state controller) Uniform State Payroll System (payroll system)

To process the State's payroll and personnel transaction documents.

Magnitude of Data

Payroll data maintained by the California State Controller's Office's Uniform State Payroll System (payroll system) for 2008 through 2012 for the following departments: California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services.

The payroll system contained more than 36.6 million payroll and personnel transactions for the period 2008 through 2012.

Purpose of Testing Data Reliability Determination

To calculate the total amount of regular and overtime pay, the average number of employees by pay type for each of the departments listed (California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services), and overtime pay as a percent of total earnings--the sum of regular and overtime pay-- for 2012.

To calculate the total amount of regular and overtime pay, and overtime pay as a percent of total earnings, by employee class title for each of the departments listed for 2012.

To calculate overtime paid as a percent of total earnings for the California Department of Developmental Services' (Developmental Services) Office of Protective Services' (OPS) employees as compared to all other department employees by location and department-wide for each year between 2008 and 2012.

For a selection of 62 Developmental Services' employees who were paid the same amount in overtime pay as regular pay for the period of 2008 through 2012, calculate the total amount of regular pay and overtime pay, and the number of employees by class title.

For a selection of eight Developmental Services' employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012, identify the employee's class title and work location, calculate the average weekly hours worked by calendar year, and total the employee's regular pay and overtime pay.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We relied on the completeness testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2012. Since the State's financial audit for fiscal year 2012-13 is still in progress, we cannot rely on this report to verify completeness between July 2012 and December 2012. However, because we found the payroll data to be complete between January 2008 and June 2012, we have reasonable assurance that the payroll data for the period of July 2012 through December 2012 are also complete. Consequently, we found the data to be complete.

We relied on the accuracy testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2011. For the period July 2011 through December 2012, we performed accuracy testing on a selection of 10 regular or overtime payroll transactions by tracing key data elements to supporting documentation and found no errors.

We performed additional accuracy testing on two key data elements—overtime hours paid and work location—for the selection of 62 Developmental Services employees by tracing key data elements to supporting documentation and found no errors.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
State Hospitals, Department of
Description of Data Agency Purpose of Data
California Department of State Hospitals' data as maintained in the California State Controller's Office's (state controller) Uniform State Payroll System (payroll system)

To process the State's payroll and personnel transaction documents.

Magnitude of Data

Payroll data maintained by the California State Controller's Office's Uniform State Payroll System (payroll system) for 2008 through 2012 for the following departments: California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services.

The payroll system contained more than 36.6 million payroll and personnel transactions for the period 2008 through 2012.

Purpose of Testing Data Reliability Determination

To calculate the total amount of regular and overtime pay, the average number of employees by pay type for each of the departments listed (California Department of Developmental Services, California Department of Corrections and Rehabilitation, California Department of State Hospitals, and the California Department of Social Services), and overtime pay as a percent of total earnings--the sum of regular and overtime pay-- for 2012.

To calculate the total amount of regular and overtime pay, and overtime pay as a percent of total earnings, by employee class title for each of the departments listed for 2012.

To calculate overtime paid as a percent of total earnings for the California Department of Developmental Services' (Developmental Services) Office of Protective Services' (OPS) employees as compared to all other department employees by location and department-wide for each year between 2008 and 2012.

For a selection of 62 Developmental Services' employees who were paid the same amount in overtime pay as regular pay for the period of 2008 through 2012, calculate the total amount of regular pay and overtime pay, and the number of employees by class title.

For a selection of eight Developmental Services' employees who were paid at least 150 percent of their regular pay in overtime for the period of 2008 through 2012, identify the employee's class title and work location, calculate the average weekly hours worked by calendar year, and total the employee's regular pay and overtime pay.

Sufficiently Reliable—We performed data-set verification procedures and electronic testing of key data elements and found no issues.

We relied on the completeness testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2012. Since the State's financial audit for fiscal year 2012-13 is still in progress, we cannot rely on this report to verify completeness between July 2012 and December 2012. However, because we found the payroll data to be complete between January 2008 and June 2012, we have reasonable assurance that the payroll data for the period of July 2012 through December 2012 are also complete. Consequently, we found the data to be complete.

We relied on the accuracy testing performed as part of the State's annual financial audit for payroll transactions between January 2008 and June 2011. For the period July 2011 through December 2012, we performed accuracy testing on a selection of 10 regular or overtime payroll transactions by tracing key data elements to supporting documentation and found no errors.

We performed additional accuracy testing on two key data elements—overtime hours paid and work location—for the selection of 62 Developmental Services employees by tracing key data elements to supporting documentation and found no errors.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A

Report type

Report type
















© 2013, California State Auditor | Privacy Policy | Conditions of Use | Download Adobe PDF Reader