Audit Results
Better Collection and Use of Data Would Enhance the Veterans Services Division’s Outreach Efforts
The California Department of Veterans Affairs’ (CalVet) Veterans Services Division (Veterans Services) has enhanced its outreach efforts to veterans and increased its use of certain data to identify gaps in service to veterans; however, it does not use all of the data at its disposal, and it does not collect other data that would help it focus its outreach efforts. In our October 2009 report, we recommended that Veterans Services use demographic data, such as the data available through the U.S. Census Bureau, as well as information that it planned to obtain from the County Veterans Service Officer programs (CVSOs) using its new electronic case management system—VetPro—to help plan its outreach efforts. Using these data, CalVet could focus its outreach and coordination efforts on counties with the highest potential for increasing veterans’ participation in federal disability compensation and pension benefits (C&P benefits). According to the manager of field operations in Veterans Services, CalVet now uses federal demographic data for veterans as a starting point for outreach. For example, Veterans Services uses data on veteran homelessness from the federal Department of Housing and Urban Development’s 2014 Annual Homeless Assessment Report to Congress to help identify the best times and places to hold stand down events for veterans. Although these efforts have merit, Veterans Services has not fully used other data available from VetPro and a new website feature we describe below that would allow it to improve targeted outreach to veterans.
In April 2014 CalVet established a feature known as myCalVet on its website, in an effort to make it easier for veterans and their families to access a variety of information, including information about state and federal benefits for veterans, advocacy providers, and employment and health care. Veterans and their family members may register and create a profile in myCalVet. These profiles can contain a variety of information about veterans, such as their age, physical location, disability rating, era of service, and whether they have dependents. Using this information, the myCalVet function provides the veteran with resources and service options based on his or her profile and expressed interests. Consequently, myCalVet is a potentially rich source of information about the attributes and needs of registered veterans.
However, more than a year after launching myCalVet, which as of May 2015 had approximately 26,000 registered users, Veterans Services’ chief of operations (chief of operations) indicated that Veterans Services has not used information collected by myCalVet to develop targeted outreach campaigns. Rather, she indicated that CalVet would prefer to have at least 50,000 registered users before Veterans Services analyzes the data and considers using them with other data for targeted outreach. Although the chief of operations noted that Veterans Services wants to use myCalVet data to evaluate the needs of veterans regionally based on their demographics and level of participation in myCalVet, she indicated that as of May 2015, Veterans Services does not have a formal plan for how it will evaluate the data in myCalVet. She stated that Veterans Services intends to finalize a plan by December 2015.
We noted that another division within CalVet has already planned to evaluate data from myCalVet to inform its outreach efforts, using data from a smaller population of registered users. Specifically, the deputy secretary of CalVet’s Division of Women Veterans Affairs (Women Veterans Affairs) indicated that as of April 2015 there were just over 2,200 women users of myCalVet, and Women Veterans Affairs is collecting demographic data on these users to help inform new strategies for targeted outreach. For example, according to the deputy secretary, Women Veterans Affairs is currently gathering demographic information on women veterans from myCalVet and other sources to redesign the content of the Women Veterans Affairs’ website by the end of the summer of 2015 in an effort to make it more relevant to registered users. In addition, the deputy secretary indicated that Women Veterans Affairs wants to use myCalVet data, among other sources, to help determine the best way to provide outreach to women veterans outside of CalVet’s Internet presence, such as social media and email. She stated that Women Veterans Affairs is considering producing a new outreach publication for women veterans, which it plans to launch by November 2015. Similarly, with a larger set of data to work with, we believe Veterans Services could use the information it has already collected from the approximately 26,000 users of myCalVet to better focus its outreach efforts. Without using existing myCalVet data, Veterans Services may miss opportunities to better target its outreach efforts to groups of veterans with similar needs that could result in more California veterans receiving available benefits.
Moreover, Veterans Services has not taken steps to evaluate how well its efforts to get veterans connected with myCalVet are working. According to the chief of operations, Veterans Services has promoted myCalVet to veterans in a variety of ways, including supplying veteran service outreach partners with postcards advertising the benefits of myCalVet to give to interested veterans, emailing veterans subscribed to its electronic mailing list about myCalVet, attempting to register veterans at events, and broadcasting a video on the CalVet website and at public events that identifies the benefits of myCalVet.
However, according to the chief of operations, Veterans Services has not analyzed the effectiveness of its outreach activities for myCalVet to determine what methods are resulting in more people registering with myCalVet, or which outreach efforts could be improved to generate more registrations. For example, veterans are required to identify how they learned about myCalVet when creating their user profiles. With such a potentially rich source of information, we expected to see that Veterans Services evaluated these data to identify trends about how the approximately 26,000 currently registered users first connected with myCalVet, and was developing strategies based on that analysis to focus its outreach to the many veterans in the State who have yet to register. However, according to the chief of operations, as of June 2015 Veterans Services has not evaluated these data to identify trends in how new users became connected with myCalVet. The chief of operations indicated that Veterans Services anticipates that it will evaluate the effectiveness of its outreach activities for myCalVet by the end of 2015 to determine what methods are resulting in more registered users. Without such an analysis, Veterans Services cannot identify the most effective processes for getting veterans registered with myCalVet and better connect them with the services they need.
We also found in our 2009 report that CalVet did not require the CVSOs to inform it of their outreach activities, limiting CalVet’s ability to identify where and how to best focus its outreach and coordination efforts. As of May 2015 CalVet still did not require the CVSOs to report information on their outreach activities, according to the chief of operations. As a result, although a statewide CVSO organization has provided CalVet with some information regarding the CVSOs’ activities, this information lacks details that could be useful to CalVet. Veterans Services could require the CVSOs to report, through VetPro, more detailed information such as how and where contacts with veterans and their families occurred; a summary of services provided; special events and activities in which the CVSOs participated; and the number of referrals the CVSOs received from local organizations or businesses, such as faith‑based organizations. Veterans Services could use this information to assess the extent to which the CVSOs are performing outreach to veterans and to help determine where and how it could better target its own outreach efforts. For example, this information could help CalVet identify counties with greater needs—such as those lacking resources to conduct adequate outreach—and position CalVet to work with the CVSOs and local interagency network coordinators in those counties to increase veterans’ awareness of available benefits and potentially increase their awards of these benefits.
According to the chief of operations, CalVet does not have the authority to require the CVSOs to report their outreach activities in such a manner because they serve the county governments. She stated that although Veterans Services wants all CVSOs to report their outreach activities in VetPro, not all CVSOs are willing to do so. Our legal counsel believes that CalVet has had the authority since at least 2009 to issue a regulation that requires the CVSOs to report information on their outreach activities to Veterans Services. Specifically, according to state law, Veterans Services may require the CVSOs to submit information, including information pertaining to their outreach activities, which CalVet needs to prepare an annual report of the CVSOs’ activities. As a result, CalVet could develop a regulation to require the CVSOs to report all data regarding their respective outreach activities through VetPro. After we shared our perspective on this issue with the chief of operations, she began working with CalVet’s Legislation and Government Relations Division and informed us that she had contacted the Office of Administrative Law about the possibility of developing a regulation requiring the CVSOs to report their outreach activities to CalVet. If adopted, such a requirement will help ensure that Veterans Services has complete information regarding the CVSOs’ outreach activities.
Our current audit determined that in December 2011 Social Services began performing monthly address comparisons and has performed this process regularly since then. However, for nearly two years, Social Services had a significant deficiency in the methodology it used for the address comparisons. It discovered this deficiency in fall 2013 and immediately corrected it. Furthermore, although Social Services now performs regular address comparisons, as we previously recommended, we found that it does not adequately track the screening or disposition of all the address matches it identifies and does not document certain investigative procedures.
Veterans Services Does Not Adequately Audit the Data Used to Determine the CVSOs’ Funding, and It Lacks Procedures for Conducting These Audits
In our 2009 report we found that Veterans Services did not require the CVSOs to provide it with information about the number of claims the CVSOs filed for C&P benefits, and we recommended that it do so. In reviewing Veterans Services’ process for auditing workload activity reports, we determined that the CVSOs now report the total number of claims to Veterans Services through VetPro. However, although state regulation requires CalVet to perform audits to authenticate each county’s reported workload activities, Veterans Services does not adequately fulfill this responsibility. Veterans Services recently recognized this shortcoming and hired an employee in September 2014 to conduct audits of workload units, which are claims processed by CVSOs that have a reasonable chance of obtaining a monetary or medical benefit for veterans, their dependents, or their survivors. The chief of operations indicated that Veterans Services currently reviews the CVSOs’ workload activity reports to identify deviations from CalVet policies that specify what activities may be claimed for reimbursement and for other errors typically found during an audit, such as a CVSO reporting the same claim as a workload unit more than once.
Although this process reveals some errors and inaccuracies in the workload activity reports, it does not detect errors that can be identified only by reviewing the records that are the basis of the reports. For example, Veterans Services’ current audit process generally does not identify claims processed by the CVSOs that did not include all required documentation, such as all of the required forms necessary for a college fee waiver. By reviewing a selection of claims processed by CVSOs to ensure that they include the appropriate documentation, Veterans Services would gain increased assurance that the CVSOs’ reported claims are valid for reimbursement. Veterans Services also compares the total number of claims the CVSOs’ report in their daily workload activity reports to the total number of claims they report on a summary form to ensure that the two totals match. This approach is not sufficient to verify the accuracy of the total number of workload claims, as the source for both numbers is the same. Because of the deficiencies in Veterans Services’ approach to verifying the accuracy of the workload activity reports and because the total number of workload units claimed by the CVSOs is the basis for the funding they receive, Veterans Services is not adequately ensuring that the funding it distributes to the CVSOs is consistent with their actual workloads.
To achieve a higher degree of assurance that the number of claims the CVSOs report in their workload activity reports is accurate, the chief of operations indicated that she intends to implement a review of selected claims, which would include selecting a number of claims, reviewing the source documentation for the claims submitted by the CVSOs, and verifying that the claims are being processed by the U.S. Department of Veterans Affairs. However, as of April 2015, Veterans Services had not formalized or implemented this additional review. The chief of operations indicated that this review will commence in the fall of 2015.
Although Veterans Services now conducts reviews of the CVSOs’ workload activity reports, it does not have documented audit procedures for doing so. The chief of operations explained that, as of April 2015, a consultant who oversees the auditing of the claims submitted by the CVSOs is in the process of training a newly hired staff person on the steps the consultant uses to conduct such audits using Veterans Services’ Procedure Manual for Subvention and Medi‑Cal Cost Avoidance Documentation as a guide. However, due to the lack of documented audit procedures, Veterans Services lacks assurance that its staff will consistently conduct workload audits correctly and completely because the processes to do so are not defined. Because the audits are an important part of the process for ensuring that funding for the CVSOs accurately reflects their workload, it is critical that they are conducted correctly, completely, and consistently. The chief of operations explained that Veterans Services intends to document its audit procedures as soon as the newly hired staff person becomes fully trained, which she anticipates will occur by the fall of 2015.
Furthermore, in part due to the fact that Veterans Services lacks documented procedures to provide guidance to staff when auditing workload activity reports, our review of selected audit files found inconsistent documentation of its audit results. Specifically, we reviewed 10 files for audits that Veterans Services conducted during 2013 and 2014 and noted that eight contained an email to the respective CVSO communicating the audit findings; the remaining two files lacked documentation regarding Veterans Services’ findings or lack thereof in the related audits. Thus, the audit findings for these two CVSOs were unclear. The chief of operations indicated that, while the missing emails were most likely sent to the CVSOs, they were inadvertently left out of the audit files. Nevertheless, not maintaining a documented record of audits could be problematic if Veterans Services were asked questions by the CVSOs regarding undocumented audits. In addition, by not maintaining such documentation, Veterans Services may miss opportunities to use information from completed audits to aid in its planning for future audits.
Veterans Services Needs to Strengthen Its Reviews of College Fee Waivers
CalVet lacks adequate oversight of the College Fee Waiver Program. Under this program, veterans’ dependents who meet the eligibility requirements may have their college tuition and mandatory fees waived if they attend California public postsecondary education institutions. According to state regulation, Veterans Services is to review and determine the eligibility and qualifications for each applicant for educational benefit. According to the CalVet deputy secretary for Veterans Services, the primary responsibility for determining whether applicants are eligible for waivers has been delegated to the CVSOs. According to the chief of operations, in fiscal year 2013–14 the CVSOs approved nearly 21,000 fee waiver applications. However, Veterans Services does not have a process to ensure that the CVSOs are accurately approving these applications. The chief of operations indicated that, as part of its audits of workload activity reports, Veterans Services determines only whether CVSOs have claimed too many workload units for the waivers; CVSOs may be reimbursed for only one college fee waiver per student per academic year.
As we noted in our 2009 report, such weak oversight places the State at risk of waiving college fees erroneously, and thus we recommended that Veterans Services take steps to meet legal requirements regarding verifying the appropriateness of college fee waivers. The chief of operations indicated that Veterans Services has only recently begun to develop procedures to perform this verification, as previously it had neither the staff nor the resources to do so. She stated that in order to provide assurance that the CVSOs are not waiving college fees inappropriately, Veterans Services’ audit staff is currently working to develop procedures for reviewing a selection of college fee waivers for each county for appropriateness. However, according to the chief of operations, Veterans Services has hired an auditor who is currently in training and will be fully functioning in the fall of 2015. By then, she explained, Veterans Services expects the procedures for reviewing college fee waivers to be documented and for reviews of fee waivers to commence and be performed on a regular basis.
Recommendations
CalVet should fully implement the recommendations from our 2009 report.
Veterans Services should develop and implement a plan by December 31, 2015, to routinely analyze and use myCalVet data to identify trends in the services veterans and their families indicate they are most interested in and incorporate the results of such analyses into its outreach efforts.
To determine the most successful methods for informing veterans and their families about the myCalVet website, and to increase the number of registered users, Veterans Services should evaluate myCalVet data to identify which marketing methods were most effective in informing registered users about the website. Veterans Services should complete this analysis on an annual basis, beginning no later than December 31, 2015.
To enhance the effectiveness of its outreach activities, CalVet should initiate, by October 31, 2015, steps to establish a regulation, in accordance with state law, that will require the CVSOs to report information on their outreach activities to CalVet.
To ensure that the funding it distributes to the CVSOs is consistent with their actual workloads, Veterans Services should, by December 31, 2015, develop and implement procedures to more thoroughly review the accuracy of the data in the CVSOs’ workload activity reports by reviewing the records that are the basis of the reports.
To ensure correctness, completeness, and consistency in its audits of the CVSOs’ workload activity reports, Veterans Services should formalize and document its audit procedures for reviewing these reports by December 31, 2015.
To improve its oversight of the College Fee Waiver Program and ensure that the CVSOs are not erroneously waiving college fees, Veterans Services should develop and implement a review process for college fee waivers by December 31, 2015.
We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the Scope and Methodology section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
Respectfully submitted,
ELAINE M. HOWLE, CPA
State Auditor
Date:
July 2, 2015
Staff:
John Billington, Audit Principal
Laura G. Kearney
Ryan Grossi, JD
Kelly Reed, MSCJ
Legal Counsel:
Scott A. Baxter, Sr. Staff Counsel
For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255.