Spilling Secrets: Secrecy & Police Data

There is a very amusing debate going on on Twitter at the moment among policing criminologists. I will skip the less than salacious details and focus on one of the charges made: police researchers have failed to exhaustively study the 'harms of policing' because of the secrecy of police, who presumably do not want to share data that will paint police services in a less than flattering light.

That charge is partially true but not in the way intended: the reality of police data is not that it is full of horrendous secrets. The big secret is that it is often a horrendous mess.

Allow me to explain. We have approximately 200+ police services in Canada. Each has their own RMS (record management system). Each RMS is proprietary, meaning that it pretty much comes out of a box as designed by the company selling it. Thus the data fields differ from police service to police service. Working on data from more than one police source is a veritable nightmare of cleaning and restructuring to create one comprehensive data set.

And that nightmare is not just for researchers. Much of the data captured by this software is incapable of easily being shared between police services, which means they have to create software portals to do so. Or, they have to run separate data collection systems and duplicate the entry and verification work. Thus, there are CAD systems, RMS systems, CPIC, there is investigative software, there is ...

No, I'm not making this up. In fact, one of my favourite examples is the RCMP. Despite being one police service across Canada, they have two major RMS systems: PRIME in B.C. and PROS everywhere else. Anyone who has ever worked in BC as a Mountie will quietly confess sotto voce: they prefer PRIME. You know who doesn't love PRIME? Other police services in B.C. that have their own RMS systems. Data in BC is shared provincially, which makes data extraction painful because the data structure is different.

This scenario is actually not all that bad considering what some police services in the U.S. deal with. There are approximately 18,000 police services, each with their own RMS systems. And some of those systems take proprietary to a whole other level. A couple of years ago, it was discovered that some smaller services in the U.S. had entered proprietary agreements that meant they did not actually own their own data. If they wanted their own data pulled, it cost them.

Guess what? I'm still only on problem #1.

Here's the thing: police data is 'fit for purpose', meaning that it is intended to aid police services in processing cases (and maybe to create some basic stats). What these systems were not developed for - and I'm not just speaking of software but the entire data collection process - was comprehensive data sharing and research. Another great example? The federal missing persons database: NCMPUR. Fantastic idea, right? A national database of missing persons and unidentified remains. Small problem: a couple of years ago, when I tried to access information on resolved cases, I was advised there is no statutory requirement for services to report closed cases. This might seem like an easy fix, but it does mean additional manual workload for every police service across Canada. Why? Because there is no software pipeline from all of the different RMS to the database.

And, yes, I'm still only on problem #1.

To fix this one issue across Canada would not only require agreement with the feds and each of the provinces, but would cost, at a conservative estimate, at least a billion dollars. We're talking about not just software and service-wide training for 200+ agencies, but also conversion of all of the old data.