Moving from "Evidence-Based" to "Hypothesis-Driven" Policy-making

February 27, 2021
Posted in:
Est. Reading Time: 4 minutes
Portion of the digital whiteboard produced through Workshop 1 brainstorming. For more information, please see the workshop webpage.
Share this article

In February 2021, the City of Hayward, Calif. convened an eight-week Policy Innovation Workshop on Community Safety with the goal of developing and testing policy ideas aimed at improving community safety. Workshop participants include community members and city staff, including representation from the city’s police force and criminal justice system. The workshop was convened in response to a letter from several city council members to Hayward’s city manager. ESAL Founder & Chair Arti Garg is one of the workshop’s community participants. This blog series is a diary of Garg's participation in the workshop. For additional entries please see:

Workshop 1 - February 18, 2021
February 19, 2021

Our first group workshop took place yesterday evening. Doing this kind of work virtually is difficult, and I was impressed with the way our city staff used small (3-4 people) breakout rooms to help build a sense of community within a large group. Much of our first two hours were spent helping participants unfamiliar with lean innovation learn more about the process and familiarizing us with the online collaboration tools we would use, such as a digital whiteboard. I’ve been using these processes and tools for years in my work as a data scientist and technologist, and I’m interested to see how their use will be similar or different in a policy innovation setting.

For the final hour of the workshop, we looked at data collected through the community conversations on public safety effort the city had undertaken last year. The data included quantitative survey results as well as selected quotes from interviews conducted with community members. I was struck by how inconclusive the data were, especially if you approached them with too narrow a scope.

For example, the survey data showed that 63% of Hayward residents feel very safe or safe interacting with the Hayward police department. Considering only that number, one might conclude that Hayward is doing well from a policing perspective. When data were further broken down by race, however, racial disparities were undeniable. 50% of residents who identify as Black feel safe interacting with Hayward police compared to 69% of residents who identify as white. Also notable was that fewer (59%) Hayward residents felt safe “in general” than they did interacting with the police, with residents who identify as Black reporting a higher general sense of safety (69%) than residents who identify as white (55%). The survey report also provided some quantitative analyses of the open-ended responses to a question about how the city could improve community safety and reform policing, with the top two responses aligning to “More police presence” (24%) and “More community involvement” (20%). In short, the quantitative survey results left a lot of open questions about what the public safety issues in Hayward might or might not be.

Reading the qualitative quotes in detail, however, patterns became more apparent. Many quotes reflected that residents do not feel heard by city government overall, not just by the police department. There were also several concerns raised about the need for the city government to directly acknowledge and address racism as an issue faced by its residents, without pointing to a specific policy change but rather to a need to feel recognized and supported. And pervading many comments was a clear sense that many city residents do not have trust in our local government.

During the workshop, the facilitators used a common lean innovation technique of asking participants to write their observations of the survey results on (virtual) sticky notes, categorizing those observations, and then asking participants to vote one which of the categories they felt were most important to address through the process. We will get the results of the votes next week, but it’s already apparent that we will be addressing the community safety question through a much broader lens than just policing reform.

After just this first workshop, it’s become clear to me that as engineers and scientists we too often oversimplify the goals of “science policy.” I often hear the benefit of people with STEM backgrounds participating in policy processes reduced to our ability to help ensure that decisions are “evidence-based.” But what does that mean when the evidence that exists is inconclusive? As a physicist, I cannot honestly argue that the process we used to decide on priorities for these workshops met the standards of analytical rigor used in the sciences. As a technologist, though, I can tell you that I’ve seen this approach produce successful outcomes by identifying unexpected solutions to problems.

So, if this supposedly data-driven approach to policy innovation doesn’t meet the criteria of statistical soundness used in science, what do we as scientists have to contribute? Perhaps we need to reframe our understanding of our role in policy discussions away from ensuring it is “evidence-based” to ensuring it is “hypothesis-driven.” As engineers and scientists, we are trained to formulate testable hypotheses for phenomena we observe that can’t be explained by existing theories or principles. Prior to this workshop, many of the ideas (experiments?) that have been proposed by Hayward community members for improving community safety have centered around policing. While those ideas will be part of the discussion, they are unlikely to address the non-policing related community safety concerns the data reveal. We may need to formulate a framework (a hypothesis?) that’s more expansive than policing to address those concerns. And, isn’t formulating informed hypotheses about phenomena we don’t understand what scientists are trained to do?

Engineers & Scientists Acting Locally (ESAL) is a non-advocacy, non-political organization. The information in this post is for general informational purposes and does not imply an endorsement by ESAL for any political candidates, businesses, or organizations mentioned herein.
Published: 02/27/21
Updated: 07/31/22
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram