One does not have to look too long or too hard to find or access one of the myriad artificial intelligence (“AI”) tools available to businesses and consumers alike that industry evangelists proclaim will change the way we work and live. And while some have clear applications we already can wrap our heads around—say, a student using an AI tool to write that term paper at the last minute—AI almost certainly has a lot more to show us still in terms of its full potential and contributions.
In light of this reality, the Legislature is working to define more clearly the appropriate use of AI tools in California’s healthcare industry. On February 13, 2024, State Senator Becker introduced SB 1120, which aims to set parameters around the use of AI tools by health plans with respect to utilization review and care delivery. The bill would accomplish this by requiring that algorithms, AI, or “other software tools” be used in a fair and equitable manner, pursuant to publicly disclosed policies, and that the use of such tools not supplant provider-driven decision making or review and consideration of a patient’s individualized medical history and clinical needs.
While there is still a long road ahead, the bill already has seen one substantive amendment: excising the requirement that health care service plans ensure that a licensed physician “supervise the use of artificial intelligence decision-making tools” whenever said tools are used to “inform decisions to approve, modify, or deny requests by providers for authorization prior to, or concurrent with, the provision of health care services to insureds.” This amendment came about in response to concerns over the type of supervision expected or required of physicians to ensure compliance with the proposed law. And because a violation of the Knox-Keene Act can expose a health care service plan to liability, removing such vague and ambiguous supervision language removes that category of compliance risk.
Notwithstanding these developments, SB 1120’s goals seem largely to follow the lead already established by CMS in its February 2024 FAQ guidance on the April 5, 2023, final rule for Medicare Advantage coverage criteria. In that guidance, CMS acknowledged the role AI tools already play in making coverage decisions, and offered industry guidance on how health plans should think about utilizing AI tools in this context. For example, CMS emphasized that AI tools not be used as a substitute for a physician’s recommendations, clinical notes, or the patient’s individualized medical history. CMS further expressed its concern that algorithms and AI tools potentially could exacerbate discrimination and bias, and admonished those who utilize these tools to be mindful of complying with the nondiscrimination requirements of Section 1557 of the Affordable Care Act. Like SB 1120, the CMS guidance also requires that AI tools be used in a manner consistent with publicly posted policies. In other words, SB 1120 appears largely to follow in CMS’ footsteps, meaning health plans might not have to worry about navigating dramatically different statutory and regulatory landscapes on this topic.
Those similarities aside, SB 1120 and the CMS guidance appear to differ in one important way: appreciating the differences among various AI tools. As noted above, SB 1120 paints AI tools with a broad brush, referring to, “algorithms, artificial intelligence, and other software tools.” CMS by contrast points out the technical differences among algorithms and machine learning AI systems. In other words, the CMS guidance acknowledges the nuance among the various technological tools on the market as opposed to SB 1120’s catchall framing. Whether this distinction will persist is yet to be seen. Setting that difference aside, California health plans can expect state and federal scrutiny into how AI tools are used to ensure they do not supplant fully the traditional approaches to patient-centered, provider-driven care.