Flock's AI Search Tool for Police: A Deep Dive into Surveillance and Oversight

Introduction

I've spent the last few months examining how law enforcement agencies across the country use Flock Safety's AI surveillance tools, particularly focusing on the company's latest search capabilities. As part of this investigation, I reviewed code that powers these systems to understand how they function in practice. What I found is troubling: while Flock has made some changes in response to criticism, many of the core design choices still leave significant gaps in oversight and accountability.

"The risk of any inaccuracies lies with the officer using the system," says Flock, but that places a heavy burden on users who may not be trained in evaluating AI outputs.

Flock's Core Capabilities

The software allows police to perform various types of searches—vehicle and person-based. For people, it accepts natural language descriptions like 'person wearing scrubs.' This is where the system gets tricky: these descriptions are run through a model that evaluates them against categories of sensitive content before determining whether to allow, block, or warn about a search.

• Standard vehicle searches use defined attributes like color, make, and model
• FreeForm person searches accept only text descriptions
• A watch list feature allows continuous monitoring of areas for matches against written descriptions

Content Moderation Systems

Flock's AI moderation system uses eight categories to flag potentially problematic searches. Categories that trigger a block include race, religion, nationality, and subjective or biased terms. Political, social, and cultural expression is the only category that issues a warning instead of a full stop.

This approach has raised concerns among privacy advocates and legal experts alike. Tom Bowman from the Center for Democracy and Technology noted that political expression is one of the most protected under the First Amendment—yet Flock's model treats it with leniency rather than strict oversight.

Additionally, the system's ability to block searches based on religion or nationality can be circumvented by rephrasing queries. For example, instead of searching for someone wearing a specific religious garment, an officer could search for 'person in white clothing' and still find matches. These workarounds highlight a critical flaw: even when systems are designed with safeguards, they can be easily gamed.

Real-World Misuse

There's no shortage of evidence that these tools are being misused. Reports from The Washington Post show at least 50 officers in the U.S. have faced charges or accusations for using Flock's cameras to track former partners, romantic interests, and others not involved in any criminal activity.

In one case, a Milwaukee officer searched for a woman he was dating 124 times and her ex-partner 55 times—logging each as an 'investigation.' Another Georgia chief ran his ex-girlfriend and her daughter 600 times before being charged and taking his own life.

These examples aren't isolated. A 2016 Associated Press investigation found over 325 similar incidents between 2013 and 2015, where police employees were fired, suspended, or forced to resign for misusing databases. In recent years, agencies like ICE and CBP have faced scrutiny for similar abuses.

Systemic Failures

The real issue lies in how Flock's technology is structured. The moderation decisions are made by AI models running on Flock's servers, unseen by local departments. That means the system can flag a search and return a warning or block—but without transparency into how those decisions were reached.

According to Deepak Kumar of UC San Diego, who studies trust and safety systems, these interfaces often fail because they don't consider both input and output. A tool that blocks or warns based solely on the initial query misses the bigger picture: what happens after a search is completed.

Kumar emphasizes that such systems function more as records than deterrents unless there's active review of those logs by supervisors and oversight bodies. Without that, the potential for abuse remains high.

Local Oversight vs. Company Responsibility

Flock argues that responsibility for how its software is used lies primarily with local police departments. While vendors can offer recommendations, it's ultimately up to individual agencies to establish policies around access and usage.

However, this decentralized approach creates a dangerous gap in accountability. When surveillance tools are deployed widely without clear standards or consistent enforcement, the likelihood of misuse increases dramatically.

Some departments have taken steps to address these issues. For instance, Audit Assistance—a feature designed to detect abnormal search patterns—is now enabled by more than one-third of Flock customers. But it remains off by default, and there's no guarantee that all agencies will activate or use it effectively.

The Future of Police Surveillance

As artificial intelligence continues to reshape public safety tools, we must ask what safeguards are truly necessary. Flock's latest system doesn't just track people—it tracks the potential for abuse, too.

We need better transparency, stronger oversight, and more robust checks on how these technologies operate. The current model, which relies heavily on AI moderation without external auditing, falls short of meeting those needs.

Lawmakers, police leadership, and tech companies alike must come together to ensure that surveillance tools serve justice, not just convenience.