Drawing Causal Inferences – Back to the Hubs

Yes, I’m about to bang on about this again. Why? Because someone recently reached out to ask me what I thought about the evidence base for this stuff.

So, I'm going to bring it up again ... and with the usual caution and trepidation, knowing I’m about to be inundated with email, DMs and comments from advocates who will insist that the hub/situation table model works because “they’ve seen it work.” By ‘work’ what they say directly, or otherwise imply, is one of two outcomes:

a. It saves or otherwise improves lives, as some advocates have directly claimed both publicly and privately to me, and/or;

b. It reduces police demand*.

Now those are two very different things. A. is an outcome and B. is an output.

If A is true, then those who make this claim because “they’ve seen it” should easily be able to back up their claims with credible data drawn from actual cases. That doesn’t happen. All I have seen to date are anecdotes on the Internet** or delivered in personal discussions.

Now about that blog title. What is a causal inference, you ask?

A casual inference is basically a statement as to why something happens or how something occurred.

In research, we generally conduct statistical modeling to explore relationships between things. However, even that type of activity can lead us astray. You need to collect your data thoughtfully, carefully control for various factors, analyze data rigorously, test your ideas with a degree of skepticism*** and subject your results to other experts in the field. What we don't do is draw conclusions from one anecdote on the Internet.

Let me show you what I mean: What if I was recently told by a credible source that at least one of the cases processed by a situation table is no longer creating police demand (B. output) because they died (A. outcome) from reasons related to one of their identified 'risk factors'. Based on this information – and using the same standards for drawing causal inferences that are employed by my friends on the Internet – could I reasonably conclude this person died because of the situation table? After all, if the “wrap around services” had worked as promised, this person should be alive, well, and on their way to thriving in the community, no? Seems pretty common sense to me!

See how that works? This type of causal inference-ing from anecdote clearly cuts both ways.

It is also why when you make claims, serious researchers ask you to back it up with rigorous evaluation research executed by highly competent, non-biased researchers who clearly document how they conducted that work and might even be willing to share the data with other credible researchers so they too can replicate the study (thus meeting several important principles of research). This is how we science.

* It's important to consider, by the way, that reducing police demand is not necessarily the same thing as reducing crime - although these two things are frequently conflated. What do I mean? It may very well be the case that individuals are no longer appearing in police calls for service because a. they're in jail; b. they haven't been caught yet; c. they moved out of province. Without outcome data, how do we know? Also, what if crime goes up? As has happened in some cities where hubs/situation tables are in use. Can we infer that means that they don't work? See that ole causual inference from anecdote and crappy research issue again.

** Do I really need to warn you about taking claims made on the Internet at face value? [insert side eye here]

*** If you don't apply a healthy degree of skepticism, you end up befouling your work with the problem of 'spurious correlation' - that is, drawing causal connections between two things that are likely caused by something else. One of my favourite examples lately is a study of Nobel Prize winners and their purported propensity to eat chocolate. The stated inference is that chocolate increases intelligence, so get to eating! Joking aside, the described effects (Nobel Prize winners) might be better explained by factors other than chocolate consumption.