When "AI for good" quietly turns into "AI for profiling"
What Bristol’s crime‑prediction experiment teaches leaders, DPOs, and data teams about trust, bias, and the human impact of “AI for good”
Reading WIRED’s investigation into Bristol’s crime‑prediction machine, I found myself thinking less about the tech and more about the people whose lives were quietly turned into risk scores.
For more than a decade, Avon and Somerset Police and Bristol City Council reportedly used the Think Family database to feed at least 23 predictive models, some aimed at spotting which children were most likely to be criminally or sexually exploited, or end up NEET. At least two child‑risk models were later halted after independent reviewers found poor accuracy and struggled to even locate the source code and variables used.
With my data protection hat on, this is the sort of story that should stop us mid‑bar and ask: “If this were our product, would we be proud of how it was used?”
Here are three lessons I’d pull out for any organisation building predictive analytics for high‑risk contexts.
Transparency isn’t a "nice to have"
The article describes years of predictive modelling layered over a large, sensitive database, yet even reviewers struggled to understand how some systems worked or how they were used in practice.
If you can’t clearly explain:
-
- what data you’re using
-
- how it’s being combined
-
- what the model is optimised for
-
- and how its outputs affect decisions,
then from a data protection perspective you’re already off‑key.
For high‑risk profiling, “trust us, we’re the experts” is insufficient.
"Genuinely poor performance" is not a footnote
Independent analysis reportedly found low precision, meaning many people were incorrectly flagged as at risk, and some results were described as “genuinely poor predictive performance”. As a black person this is particularly worrying given the bias we already know exists in such systems.
In a commercial data science setting, I think it’s a failed experiment. In policing and social care, those false positives and false negatives land on real families in the form of extra scrutiny, missed support, or stigma.
From a data protection standpoint:
-
- bad or biased data in
-
- plus opaque models
-
- plus weak governance
is not “innovation”, it’s a liability; both ethically and legally.
PoliceAI raises the stakes for everyone
The UK government has now launched PoliceAI, a national centre for AI in policing backed by tens of millions over several years, with a remit to identify, test and scale AI tools across all forces.
That means:
- more demand for analytics and AI from public bodies
- more vendors pitching “responsible AI” solutions
- and more profiling of people who often have limited ability to opt out or challenge decisions.
If you’re a data science or AI company eyeing this space, please recognise that you’re helping set the tempo for how AI shows up in people’s lives – for good, or not.
So, what does that mean in practice?
- Prepare: treat high‑risk profiling as such. Do thorough Data Protection Impact Assessments (DPIAs), map power imbalances, and design for challenge and redress from day one.
- Participate: bring domain experts, communities, and ethics voices into the rehearsal, not just the launch.
- Probe: keep testing for bias, drift, and unintended harm, and be willing to retire models that fail, even if they looked clever on the whiteboard.
If we want AI in policing and social care to sound more like a well‑conducted ensemble than a chaotic improv, data science companies and DPOs need to be part of the arrangement from the start and not a compliance check at the end.