This Above All Else: Rigor

Let's tackle something head on: those of you who have felt that evidence based policing is just the newest policing fad are correct. Yes, you're correct. As much as it pains me to say this, it HAS become the buzz term du jour. Everyone's doing it. Especially those who aren't doing it at all. But they say they do, so that should count for something, right? Wrong. Evidence based policing isn't about doing a few carefully selected interviews or cherry picking the research literature to support your predetermined conclusions - although I've seen both held up as examples of EBP in Canada. It's about:

There's several important words here, but the one I wish to hone in on is: rigor, as in policing research should be rigorous.

Much of the research being done is anything but rigorous*. In fact, in some cases, the research is non-existent. If you point this out, people get mad at you. Very mad. I'm okay with that. CAN-SEBP is not about winning a popularity contest and I never aspired to be a prom Queen. CAN-SEBP is about setting a national research agenda that prioritizes scientific rigor above picking hot topics, making friends, appealing to everyone (and therefore appealing to no one) or just otherwise going along to get invited into the club. What does scientific rigor look like? It looks a lot like this (see 4-5) **:

Where is Canadian EBP at? Outside of some small academic circles in places like Carleton and Simon Fraser, and in a handful of police agencies***, our output tends to mostly hover somewhere around 0 to 1. When’s the last time you were at a Canadian policing conference hearing about a series of RCTs, longitudinal studies, meta-analysis, multi-site comparative studies with treatment and control groups? Now, let’s compare our situation with that of the UK: love them or hate them, randomized controlled trials (level 5) are a frequent feature of the work presented, and, as a result of setting the bar high, SEBP, the Cambridge program and other groups have helped to force UK police to at least consider a higher standard of evidence than what is routinely accepted here as gospel-level proof. The UK also has - relatively speaking - a diverse array of other forms of research being done, including some strong qualitative work, which is presented at CCCU every year. Same with the U.S.; police officers are presenting research that would score a level 3 to 5. You see these pracademics annually in pockets of work presented at IACP and definitely at ASEBP. What have I seen here by contrast? Descriptions of programs with promises of future study. Output evaluations being used to infer causality (outcomes). Studies with about 14 interviews used to make policy recommendations about changing the criminal justice system. Lit reviews in which the author openly admits they selected only the studies with the results they wanted. Don't even get me started on some of the material that gets presented to police practitioners at conferences. It would be easy to ignore all of this and just go along with watching everyone jump on the EBP train as it slowly derails, but I just can't do that. It's not what I signed up for. And for those unhappy with the contents of this blog, I have one thing to say, don’t bitch, do better.

* I’m not talking about error-free. Mistakes happen. One just happened to me. And that fact - that mistakes happen despite best efforts and that our work will always be subject to some error and some limitations - is why we publish our methods and results and people comment. Then you correct them and improve. That’s how science works. What it doesn't do is say, 'we have evidence, the best evidence, we know all the best evidence, but we can't show you any of our very strong evidence, but it's the best.’

** Thanks to Jerry Ratcliffe for this. See: http://www.jratcliffe.net/blog/not-all-evidence-is-created-equally-an-update/

*** There are a number of police practitioners and crime analysts working individually, collectively, with academics and/or within their police services to produce quality work. It's important not to forget that. I wish we heard from them more.