Evaluating qualitative research - an idea
I was recently asked to work on a review of the existing Canadian literature on missing persons. In theory, this should be an easy task: there isn’t much.
That’s the problem: there isn’t much and, what is there, is a tad … not good.
What’s a girl to do?
I decided to deal with this problem in two ways. First, I would run something called a scoping review. A scoping review allows you to map out what is and is not available in the research literature. Because the peer-reviewed, published research is so limited, I had to break down and include some of the ‘grey literature’ (the non-peer-reviewed, non-published work*).
The second thing I decided to do was to evaluate the available research based on its possible use to inform evidence-based policy. This is actually an equally important task for one simple reason: because of the lack of high quality research in many areas of Canadian criminal justice, we sometimes see real dreck submitted to federal inquiries, cited in provincial reviews and commissions, held up by policy-makers as ‘best practices’ and used to support public policies, programs and practices that, quite frankly, have zero credible evidence base. Why?
Three (non-mutually exclusive) answers come to mind:
The cynical view – as Paul Rock documented a long time ago in his work on the Solicitor General’s Crime Victims legislation, policymakers cherry pick experts, opinions and so on that cohere with their pre-existing views and policy goals;
The ignorant view – policy mavens simply don’t know (and don’t have the staff to tell them) that some junky report is actually not based on anything approaching solid research;
The ‘we don’t have anything else, so let’s use this’ view – policymaking must have the appearance of being consultative, therefore, in the absence of anything else, policy makers gravitate to what they have at hand.
I can’t do much about #1. And #3 I’ve been working on for four long years. But maybe I can do a little something about #2 and that’s why I decided to tackle the issue of what constitutes halfway decent qualitative research. You see, a lot of the research on missing persons is qualitative (based on interviews, documents and, in one instance, billboard signs). How do you assess the quality of a study of billboard signs?
So, here’s what I’m working on: a checklist to assess qualitative research based on work done by the Critical Assessment Skills Program (CASP) for evaluating research in healthcare. This is a work-in-progress. Future iterations will likely include questions on policy-relevance (and other stuff).
*Don’t bother sending me anti-academic hate-mail because I don’t care. The bottom line – and I’ve said it before – is that a lot of the grey literature on crime and criminal justice topics is terrible. Does that mean academic peer-reviewed work is all grand? No. And if you sit with me for a couple of hours in EBP training, you’ll watch me shred a lot of it. The peer-reviewed literature just tends – overall – to have some much better work in it.
