Report 2020-104 Recommendation 8 Responses

Report 2020-104: Calbright College: It Must Take Immediate Corrective Action to Accomplish Its Mission to Provide Underserved Californians With Access to Higher Education (Release Date: May 2021)

Recommendation #8 To: Calbright College

To ensure that it adequately prepares its target student population to obtain positive employment outcomes after graduation, Calbright should, by November 2021, develop and implement a process for selecting and expanding educational programs that will provide value to that population; that process should include the following:

1. Collaboration with employers and industry groups to inform the content of the programs.

2. Consideration of market demand for graduates of such programs.

3. Determination of whether the programs can help its target student population obtain positive employment outcomes including jobs, earning gains, and upward mobility.

4. Available resources for program implementation.

5. An evaluation of student demand for the programs.

Annual Follow-Up Agency Response From October 2023

As noted in previous updates, Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The auditor's office has shared that it is satisfied with the information provided in past updates and has made an additional request for Calbright to provide its Standard Operating Procedure (SOP) that outlines the College's process for identifying, assessing, and selecting potential programs. The College had previously (in November 2022) provided a draft version of this SOP, and then received further guidance noting that a final version was necessary in order to complete this recommendation. A final version of the SOP, which includes steps the College takes to review available resources for the program (such as technology infrastructure, staffing, and licensing costs), was provided to the auditor's office in October 2023. This SOP will continue to be updated throughout the College's start-up period in order to reflect evolution in Calbright's development and operations.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Calbright provided its updated procedures for selecting educational programs, which now includes that Calbright will identify the costs associated with the educational program. It further includes as criteria for proceeding with a new program that Calbright has sufficient financial resources available for the program.

Annual Follow-Up Agency Response From November 2022

Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The auditor's office has shared that it is satisfied with the information provided in past updates and has made an additional request for Calbright to provide its Standard Operating Procedure (SOP) that outlines the College's process for identifying, assessing, and selecting potential programs. The auditor's office has indicated that this recommendation will be fully implemented after the SOP is provided. The SOP was shared with the auditor's office on November 7, 2022.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Calbright provided its draft procedures for selecting educational programs. Those procedures include performing market research, identifying employer partnerships, and considering the benefits that a program will provide to Calbright's target student population. However, the procedures are still in draft form, and they do not establish a process for verifying that sufficient financial resources are available to establish the program. We will consider this recommendation to be fully implemented when Calbright finalizes a set of procedures that address all aspects of our recommendation, including verifying that resources are available to launch a program.

1-Year Agency Response

Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The College's evolution in this area has led to the development of a workforce-driven equity-centered design approach for program selection which acknowledges that equity requires intent and focus.

The College's decision making leverages labor market insights to assess traditional supply and demand metrics as well as in-demand occupational skills, including workforce analysis to gauge hiring probability of our target population. We utilize data sources such as EMSI to inform program selection, including information on total job postings in California for roles related to specific programs; regional and state-wide demand for these roles; expected job growth within California over the next decade; and whether the median and range of salaries for those positions will foster economic mobility. In addition, we maintain an Advisory Council for each program area composed of external members who provide expert feedback and collaborate to help identify in-demand jobs and the technical and soft skills they require. Research on job postings' educational attainment requirements, as well as race equity within incumbent workforce by occupation, further validate that the College's programs will enhance both job prospects and economic mobility of our target population.

We are creating market demand by responding to critical gaps in the workforce that have historically been the responsibility of the employer to address. Our recent partnership with SEIU-UHW to develop a Licensed Vocational Nurse training program is one such example of this new approach. We are working in tandem with the labor organization to address a need felt acutely in every region of the state, expand the training pipeline, and diversify the frontline healthcare provider workforce.

California State Auditor's Assessment of 1-Year Status: Pending

Calbright provided documentation containing some information on factors it considers when researching new educational programs. However, it has not yet developed a clear, specific, and documented process describing the criteria that it will use to select new programs or the procedures it will follow.

6-Month Agency Response

Calbright has developed and implemented a process for selecting and expanding educational/career pathways of value. This process begins with determining labor/workforce needs based on employment supply and demand, as well as available Calbright resources to expand programs, and includes quantitative validation as well as engagement with employers, advisory councils, regional workforce development boards, and worker associations to understand urgent needs and anticipate future opportunities.

Calbright maintains an Advisory Council for each program area composed of external members who, through regular meetings, provide expert feedback and collaborate to help identify in-demand jobs and the technical and soft skills they require. To date Calbright has engaged with industry leaders, employers, and advocacy groups including from the technology and healthcare sectors to both inform and validate our program pathways and career supports. To drive stronger outcomes for our students, we have also built partnerships with regional organizations and workforce development boards to align efforts to support the college's focus population.

The College utilizes data sources such as Burning Glass Labor Insights and Emsi to inform program selection, including information on total job postings in California for roles related to specific programs; regional and state-wide demand for these roles; expected job growth within California over the next decade; and whether the median and range of salaries for those positions will foster economic mobility. The College also conducts qualitative research interviews with individuals currently employed in these roles to identify pain points and key skills for their positions, as well as their career goals and aspirations.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

Calbright provided a document that it stated describes its process for selecting and expanding programs. The document provides a very high level overview of its process for selecting and launching new programs. It describes in greater detail Calbright's efforts to conduct market research and collaborate with labor market representatives. However, it provides little detail on how it will evaluate the appropriateness of potential programs for its target student population, whether the programs will provide positive employment outcomes for those students, and whether Calbright possesses sufficient resources to launch, maintain, and expand the program.

60-Day Agency Response

Calbright's process for selecting and expanding educational/career pathways starts with determining need based on employment supply and demand, and includes engagement with employers, Advisory Councils, Workforce Boards and worker associations to understand urgent needs and anticipate future opportunities. Calbright maintains an Advisory Council for each program area composed of external members who provide expert feedback. To date Calbright has engaged with a variety of industry and employers from technology and healthcare to both inform and validate our program pathways and career supports. Designing effective programs also requires us to understand employees, their environments and what makes them successful. To drive stronger outcomes for our students we have also built partnerships with regional organizations and workforce boards to align efforts to support the college's focus population.

We also conduct Labor market research to: align academic programs to support employment in high-demand areas; determine total job postings in California for relevant job roles; understand statewide and regional demand; identify top defining and distinguishing skills for job roles (to align with our curriculum); analyze the ratio of job postings to eligible graduates who could fill them; and assess salary data leading to improved economic mobility.

Calbright's process also includes determining need based on areas where our targeted student population does not have access to programming, yet there is a high demand for such a program. We also conduct qualitative interviews to glean first-hand insights into the needs of those currently in the workforce.

We have begun to document this process and are on schedule for full implementation by November 2021. Reflected in the attached documentation are specific examples for how the data and analysis occurs for a given program prior to Calbright Board of Trustees approval.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2020-104

Agency responses received are posted verbatim.