Author’s note: After this article was written, California State Governor Gavin Newsom vetoed SB 1047 on 29 September. In a letter to the California State Senate, he cited such reasons as a false sense of security, a threat to small business innovation, and a lack of adaptability.
California is at the forefront of AI regulation with the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, Senate Bill (SB) 1047. This landmark proposal aims to ensure the safe and responsible development of large-scale artificial intelligence systems (AI). In a recent virtual event hosted by the Carnegie Endowment for International Peace (Carnegie), experts discussed the potential impacts of this bill on both innovation and safety in the tech sector. Moderated by the founding director of Carnegie California Ian Klaus, the panel explored the pros and cons of SB 1047, with the debate highlighting the complexity of AI governance.
Public Opinion and Concerns
A recent survey of Californians reflected mixed sentiments: 50% of respondents expressed concerns about AI’s risks, while 35% were optimistic about its benefits. This tension underscores the ongoing debate between prioritizing safety and fostering innovation. SB 1047, which seeks to establish safety protocols for frontier AI models, addresses this balance. The legislation would require developers of advanced AI systems to follow strict safeguards, including cybersecurity measures and whistleblower protections, to prevent harms such as cyberattacks or infrastructure damage.
Support for SB 1047
Proponents like Dan Hendrycks, director at the Center for AI Safety, argue that SB 1047 is crucial and timely, a claim that is supported by leading AI researchers such as Geoffrey Hinton and Yoshua Bengio. Hendrycks emphasized that the bill targets only the largest companies which have significant resources, ensuring they implement safety measures without stifling smaller startups. Ketan Ramakrishnan, associate professor at Yale Law School, added that the bill builds on tort law and incentivizes AI developers to explore risks while offering transparency and robust whistleblower protections. He explained: “If these risks are real and serious and around the corner, then the idea that we should simply wait, muddle along, without incremental clarification of the existing law…that doesn’t make any sense.” He added, “What this bill does is it facilitates what is already going on in the industry [...] understanding risks […] and being careful before we plunge ahead.”
Opposition Voices
On the other side, critics like Ion Stoica, co-founder of Databricks, voiced concerns that the bill introduces additional liabilities for all developers, not just the largest companies. Stoica warned that placing thresholds on AI development could slow down innovation, especially in open-source projects. Lauren Wagner, an adviser to the Data & Trust Alliance representing opposition from startups and tech firms, argued that the legislation is premature, pointing out that the European Union took years to pass its AI regulations, and more time is needed to fully understand the impact of such laws. Wagner argued, “Major companies have come out against SB 1047, Open AI, hundreds of startups […] There’s too much uncertainty and we’re too early in the process.”
The Path Forward
Despite these differing viewpoints, panelist Jon Bateman from Carnegie observed that the debate feels familiar, echoing past regulatory battles in other industries. He suggested that while AI is a unique technology, the core challenge remains the same: how to regulate without stifling progress. SB 1047 is seen as a pivotal moment in shaping AI policy in the U.S. and could set a precedent for international standards. “AI is unique, but it's also in another respect just another software, and software as a whole is largely a fairly unregulated domain,” said Bateman. He ended by encouraging policymakers to “embrace the uncertainty of this moment and legislate or not legislate in a flexible manner.”
With bipartisan support, SB 1047 is on track to pass in California, potentially influencing AI regulation nationwide and internationally. The bill, while divisive, represents a critical effort to ensure AI development remains safe without stifling innovation in this rapidly evolving field.
