SIO266: Why Sam Harris Is Wrong about Police Killings

Transcript

This transcript was generated automatically and may contain errors — the audio is the record.

0:00:29 Thomas: Hello, and welcome to Serious Inquiries Only. This is episode 266. I'm Thomas, and back with me is Lindsey Osterman. How you doing?

0:00:37 Lindsey: Really, really good, yeah. Everything in my house is nice. That's all I know about, but I agree.

0:00:38 Thomas: Yep. I know, we, you know, that question should be stricken from the record in 2020. Every podcast has to do it, but I should just say salutations. Hello. Greetings. Ah, you are on the show. Neutral statements. Yeah. Yeah. Okay.

0:00:52 Lindsey: Indeed. Greetings to you, sir. You are factually accurate. Yeah.

0:01:01 Thomas: So how do we explain what we're trying to do here? Well, as I said, back when Lindsay first started coming on the show, well, not first, first, but started coming on the show, I don't know, months ago now, we had so many things we wanted to talk about, so many ways that I think her scientific expertise could really help us break some things down. And this is one of those things.

Things, um, many listeners to the show likely used to be, or maybe even still are listeners to the Sam Harris podcast. He used to be a big influence on me back, I don't know, eight years ago, six years ago, uh, until in my opinion, he kind of lost it. One of those things.

If you heard that episode where he tried to do, it's it's called "Can We Pull Back From the Brink," and it was, I, well, I'm gonna save my thoughts on it overall, but the point is, within that episode, and it's not just Sam Harris, this isn't just a reply to Sam Harris, because within the conservative kind of ecosystem, there was a response to the George Floyd protest, to the Black Lives Matter protests, that I know we've all aged 70 years in the last year, but do

0:02:07 Lindsey: Mm-hmm.

0:02:07 Thomas: You remember when we had only aged about 35 years inthe last year? Sometime in June when we were all, I don't know, in our late 70s.

0:02:11 Lindsey: Yeah, I remember.

0:02:18 Thomas: Those protests obviously were a huge deal. And in fact, side note, I even said at the time, there's no way we're going to be able to maintain all this energy long enough to actually drive the reform that we need. And it's not any fault of ours. There's too many things. So anyway, get on uncovering all this onion layer of things here. Um, there is some research that Sam Harris and those, uh, people coming from kind of, uh... anyway, um there are many people who are very anti black lives matter. And one of the things that gets cited, some of the stuff that gets cited is this research that you've looked into that you know, that has the maybe surprising conclusion that wouldn't you know it, actually black people aren't killed by police officers as much or when you adjust for certain variables. So Lindsay is here, I hope, to maybe tell us why those aren't accurate.

0:02:29 Lindsey: Yeah. Yes. Too, yeah, because one thing that everybody agrees on, regardless of, you know, whether they're in the same camp as Sam Harris or, you know, support Black Lives Matter, everybody agrees that, um, black people, as a function of the population in the U.S., are killed disproportionately by police officers. That's not something that anybody disagrees on, um, but the... the issue that has come up, and the way that the people, the researchers that we're going to be talking about today, make this claim, is that population isn't the appropriate benchmark to be judging whether there's disproportionality and how often black versus white people are being killed by police. And so Sam Harris has talked about this, and Owens has talked about this, a ton of people have trotted out findings from these articles that we're going to be talking about today to support that claim. And it sounds very convincing if you don't know what's wrong with the math that they did.

0:04:05 Thomas: Right. So that's the talking point you hear a ton from a a lot of anti conservatives, anti black lives matter people. It's well, look, sure. There may be disproportionate killings or whatever arrests or whatever you want to say, but black people are responsible for X amount of the crime. So therefore, so that's how the claim goes every time. Um, I want to do a side note for the process of researching this. We both had to listen to a bunch of Sam Harris. I didn't want, I don't want this to be a replying to Sam Harris episode.

However, I have pages of notes replying to Sam Harris because it was so frustrating to listen to that. So what think we we're going to do is maybe after this, we'll do a like a patron only bonus episode where we more specifically address those Sam Harris episodes. But this we're kind of hoping to just be more, you know, not directly a response to him, but responding to some of these claims because they're being made everywhere. I mean, this is a Fox News claim too. This is a broad problem when it comes to the disagreements on police reform.

0:04:07 Lindsey: Exactly, exactly.

0:04:35 Lindsey: Nope.

0:05:08 Lindsey: Yeah, exactly, and I really, you know, I think it's important to be able to say more in response to those things than just like, well, that's probably not true.

0:05:17 Lindsey: Like, that sounds wrong, because it does sound wrong, but yeah, Yeah, I think it's important to dig into. So these, these papers have been critiqued in very technical ways. And so I think you're gonna have to keep me honest here in terms of, yeah, how clear it is. But, but I think this will be, I really hope this will be helpful for people.

0:05:27 Thomas: I'll try. All right.

0:05:34 Lindsey: So, so yeah, so we're going to talk about the claims of two very widely cited studies that examine some version of these claims about how actually, you know, black people are not killed by police at higher rates. And, and if anything, that white people are killed at higher rates by police. So the first article is by Johnson, Tress, Burkle, Taylor, and Cesario. It was published in 2019, and the uh, the title here is "Officer Characteristics and Racial Disparities in Fatal Officer-Involved Shootings." It was cited 45 times, and it's only, it was only out for like a year before it was retracted, so that's a lot, that's, that's just Within, yeah.

0:06:10 Thomas: Spoiler, it was retracted, everybody. I wasn't sure when you were going to say that. Yeah, that's a big deal that maybe if people are still referencing this this retracted paper, that's could be a problem.

0:06:16 Lindsey: Indeed. Indeed. Yeah. So it was cited 45 times alone just in the academic literature. That's massive for only being out for a year, right? But in the popular press, this has been cited a ton, right, you know, YouTube personalities and things talk about this all the time. So, yeah,

0:06:36 Thomas: Yeah. Well, side note, this is what is so weird in kind of doing the thing we do here on a podcast. If you're on the left, which is the right has this ecosystem that's very, I don't know what to say, like centralized or something. Once they get on a thing, once they're like, oh, this paper shows this, they all do it. Like they instantly All adopt that thing. And so it goes from, you know, this obscure academic paper possibly to, oh, my God, this is being cited everywhere all the time. It's very weird.

0:06:59 Lindsey: Yeah.

0:07:10 Lindsey: Yeah, it is. It is. And yeah, it's a problem. Problem. Yeah, so this is one of the ones that Harris referenced explicitly in that episode, "Can We Pull Back from the Brink?" And it's been criticized on at least two grounds. And those are the two critiques we're going to talk about today. So the first, the most widely cited claim here that I want to criticize is that in this paper, they claim that they found no evidence of anti-Black or anti-Hispanic bias in fatal officer-involved shootings.

And they claim that they found that white officers were not more likely to shoot minorities than non-white officers, so we looked at officer race to predict the race of the person being shot, so that's the first claim that we want to talk about here. The second claim that we wanna talk about is this claim that they made that, and I'm gonna quote here, they said that a person fatally shot by the police was more than more than six times, so 6.67 times less likely to be Black than white in a typical shooting and a typical officer-involved shooting, right? Those sound like similar claims, but there's, we're to critique them in different ways, right? Okay, so those, that's where we're going, those are the two claims we're gonna try to debunk here. So I want to start with a little bit of information about the article itself and the way they got their data and things, because I think that's important to critiquing this.

So, um, open up their article by talking about, you know, recent high-profile shootings of Black Americans. This was pre-George Floyd, by the way. Um, and they suggest that it's, it's very important, very central to interpreting these cases, as to whether Black civilians are really overrepresented in fatal officer-involved shootings, and whether racial disparities are due to discrimination by white officers, right? Right?

0:08:58 Lindsey: Saying that these are the central things that people care about when they're talking about these cases. And like, those seem like reasonable claims, right? I mean, I think that those are probably central things that people are talking about. Are minorities being discriminated against and is this because of racism on the part of cops, right?"

0:09:17 Thomas: Yeah, that's it. It opens up a can of worms already. Ready.

0:09:21 Lindsey: Know, I know. I wanted to say the same thing. Yeah. Because I think that most people who are informed about this issue have concerns that are way past the bad apples argument. It's definitely not just about a few racist cops.

0:09:21 Thomas: Yeah.

0:09:37 Thomas: And I have noticed that with the conservative side of this argument and in all these things, there's an obsession, an obsession with what were the actual individual motives of the person while they were doing the thing. If they do it with Trump, they do with it. Oh, isn't there? What if there's some other explanation for why he, you know, I even had this argument with my family at one point, you know, there's always something other than racism that explains everything. And you know, to the black man who's being murdered by someone putting their knee on his neck, I don't think it matters whether or not that individual officer is like, "Yes, I am trying to be a racist right now," and that's the big problem.

So that's one of my biggest, but that's kind of a macro level criticism on this conversation. I don't know how it applies exactly to these studies. I mean, it might also be interesting to know if there's a individual bias problem. You know, it's not as though we shouldn't be looking at that, but that if you, if you even if you proved, which you may well be able to, that there's, you know, almost no individual officer bias in individual altercations that would

0:09:44 Lindsey: Yes. Absolutely.

0:10:12 Lindsey: Doesn't matter. Yeah, it does. Yes, no,

0:10:49 Thomas: Not at all solve the problem.

0:10:51 Lindsey: I totally agree. And actually, that is one of my central critiques of this paper. I mean, it's not one of the statistical ones, but they do this move that I see a lot, and I feel like Sam Harris did a little bit, which is like, all right, we're going to attack this one piece that would be relevant to interpersonal racism. And then when we don't find support for it, if we don't find support for it. We're going to say, okay, so, you know, the more important thing is just to try to get Black people to stop being such criminals all the time. And that's not an appropriate move. I mean, obviously, it's not an appropriate move. But they do it. They do it in a very subtle way.

0:10:55 Thomas: Okay. Okay. Yeah.

0:11:28 Thomas: And not to mention, this is why it's systemic. And I know you probably hear that term a lot, everybody. And it really does capture it because it's not any one thing. And that's what makes it so difficult. And what frustrates me about, and I know we're going to get more to it later, but in the Sam Harris conversation, he makes reference, passing reference to, you know, oh, income inequality, wealth inequality, big problem. But It's like, well, yeah, but that's all part of it.

Like, if If you've got a a subset of your population that is just dramatically less wealthy than the other set, you know, like, do you not think that's going to cause problems, which then might lead to over policing, which then might, you know, lead to more of these altercations, which then has the, uh, you know, statistically it's going gonna lead to them going wrong. It does, it may not mean, you not that we can say for sure either way. Well, you'll tell me what the data says, but it may not mean that like individual motivated racism is the biggest issue. It may not mean that but there's still a huge problem.

0:11:33 Lindsey: Mm hmm. Yeah. Yeah, absolutely. Sure. Yeah absolutely. And I mean he may say that he agrees with all that. But you know, another problem that I had with that episode is that like, even if he does agree with all that his emphasis is somewhere completely different.

0:12:40 Thomas: Yeah, well, okay, we're gonna don't don't we can't get started on that. We're gonna do it later. We'RE gonna just absolutely tear that episode a new one at a later date. But let's get back to the study. Sorry.

0:12:42 Lindsey: Yes. Okay. All right. All all right, cool. Okay. Okay, so they are talking about this broad problem, and then they focus it on this very specific claim, right? So the central question that sort of is guiding their research is, what is the best way to really look at racial disparities in policing? So should we be looking at how many black versus white people are killed as a function of total population? Or is there a more precise benchmark that we could use, right, so, they call this, like, the benchmarking problem. And so they, you know, they acknowledge the past research that has used population as the benchmark has showed that black people are killed at higher rates proportionally to population.

But their critique is that this calculation doesn't account for the number of encounters between police and civilians by race. So it could just be that encounters are higher for black civilians and that as a rate of encounters rather than total population, fatalities are equivalent by race, or maybe you see the reverse or something. Now, spoiler, they don't actually have information about police encounters in this, in this paper, right? And they do this in the other one too, they make this case about like, what really, what we need is encounters, and they don't look

0:13:40 Thomas: Wow.

0:14:00 Lindsey: At encounters.

0:14:00 Thomas: And they don't have it. Well, I think that's a big problem here. With all sides of this, even people who are trying to get the stats, you know, and aren't trying to just do racist conservative talking points. We don't have the data on this. We might have decent data on killings, because I know, especially now, like a lot of newspapers, a lot OF websites are like tracking this and like trying to, you Know, make sure none of those names fall through the gaps. But in terms of just encounters, we really do not have that data.

0:14:01 Lindsey: Yeah. Yes. Yes. No.

0:14:34 Lindsey: We don't, um, and and especially not in a way that wouldn't be biased from the outset, right? So we can come back to that. I've got, there's a few notes I have about like, here's what some methods could look like that actually could be better in getting us to that, but anyway. But yeah, so they don't have information about encounters, and the fact that they are talking about encounters here is just a way of like shoehorning in a claim later that Black Americans get killed more because they're violent criminals, right? So they're going to use a proxy that is not encounters, encounters, to predict the race of shooting victims, and it just, it doesn't work anyway. Yeah, so anyway, so they're making the point that the conclusion that you draw about racial disparities really depends on the benchmark that you use, and that's absolutely true. Sure it does, um, and so Johnson and colleagues suggest that one remedy to the benchmark problem is to use regression analyses to predict the race of the civilian that is shot in these officer-involved shootings, using various variables that could account for racial disparities.

0:14:38 Thomas: Right. Right?

0:15:36 Lindsey: So specifically, in this study, they predict within a database of all the civilians killed by police fire in 2015, the race of the person shot, using information about the race of the officer, plus a few other officer characteristics, details about the civilian who was shot, such as whether they were armed, whether they were attacking police, um, and

0:15:58 Thomas: Again, all of which are according to the police officers. Yeah. Isn't it? I mean, every single time we have video of a shooting that's just objectively terrible, a killing that's objectively terrible. What it will go down as according to police is, oh, well, he had what looked like a gun or he was, I was in danger.

0:15:58 Lindsey: Exactly. There's so much wrong, um, Yeah.

0:16:20 Lindsey: Exactly.

0:16:21 Thomas: And then it never nothing ever happens to that story until there's protests, until there's widespread attention on it. And then all of sudden we get the video or we get The Facts and things change.

0:16:21 Lindsey: He was resisting arrest, Exactly. Yeah. Yeah, exactly, um, and actually we're going to come back to those, those types of shootings in the second study that we talk about, so it'll be fun, um, all right, so that's what they're trying to do, they're, time to predict the race of civilians who are shot based on these different characteristics, so, so that was, yeah,

0:16:38 Thomas: Let me let me clarify. So when you're saying predict your. So we already have that data, but they're trying to, like, do a prediction and then match it up. Is that what you're seeing or to see where the discrepancies are? Okay.

0:16:48 Lindsey: Yeah.

0:16:57 Lindsey: Essentially, right, I'll I'll describe the the analytical strategy in a little bit later, but they're using they're using regression analyses, and you're essentially, like, trying to calculate predicted values for some outcome on the basis of predictor variables, um, yeah, so I'll come back to that, that's, that's, a we're gonna have to talk too much about regression actually, let's, let's not hasten that, um, but, uh, but yeah, okay, so, so they used this database of fatal officer-involved shootings, um, it was originally actually collected by the Washington Post and the Guardian, and then they added to it, um, so all fatal officer involved shootings in the U.S. in 2015, they limited their analyses to, uh, people who were shot that were white, black, or Hispanic.

0:17:03 Thomas: Okay. Okay. Yeah.

0:17:45 Lindsey: And across all of those cases, there were something like 900 fatal shootings during 2015. In order to supplement missing data that weren't there from the original database collected by the Washington Post and stuff, they actually called 684 police departments and requested info on the race, sex, and years of experience of the officers involved, which, I mean, the one good thing I can say about the study is that that's a pain in the the ass process, and they ended up with the database that I think you could actually do some interesting things with, but anyway, so they received information back from those departments and got complete data on 62% of the shootings from that, and then for the miss, for the day that was still missing, they went out to police, or to, uh, like, newspaper articles and things, and court documents, and added more to the data set, so when all was said and done, they had complete officer information for 72% of the shootings in the set, and partial information on the officers for 96% of the shootings.

0:18:45 Lindsey: So that's pretty good. That's decently complete data set. Okay, so that's their data set, before I go any further, I do want to underline one thing, right? So again, they started with this very broad context, people are getting shot more by police, why is that happening? And then it turns out that they're actually testing this incredibly narrow question of whether officer characteristics impact racial disparities only among people who have been fatally shot. And the reason I want to underline this is because the conclusions that they ultimately draw do not describe what they are actually testing. This research is not looking at the risk of being fatally shot if you have an encounter with police, or even the risk of dying if you are shot by police. It's analyzing the racial makeup of a group of people who all were fatally shot by police officers.

0:19:32 Lindsey: And that brings us to the first critique already, even before we've seen the data. So there was a response to this paper published by Knox and Mumolo in 2020. And they pointed out that given that they're only investigating the selected pool of cases, right, incidents in which an officer shot someone and they died, the possible inferences that you can draw here are super limited. Like you can't make a claim about whether white officers are more likely to shoot minority civilians because you don't have that information, right? You would need information about encounters in order to make that claim, right.

0:20:06 Thomas: Mm-hmm.

0:20:06 Lindsey: That's, that's essentially a claim about rates and you can't make a rate unless you have the, the denominator there, right, so the way that they have framed their conclusions is, is not matching up with the data that they actually have, okay. I'm actually, I'm going to quote their, their conclusion here, uh, just to show that I'm not making this up, so, uh, this is the second paragraph of their discussion section, and they say, we did not find evidence for anti-black or anti-Hispanic disparity in police use of force across all fatal shootings, and if anything, we found anti-white disparities when controlling for race-specific crime, right, so if I'm a journalist and I read that statement, what do I infer? White people are more likely to be shot by police, right, right?

0:20:09 Thomas: Right.

0:20:51 Thomas: Yeah, that

0:20:52 Lindsey: That's not, yes, it clearly seems to be what they're saying. And like, they made a response to this where they were like, oh, no, this was just misinterpretation. But like, no. Sorry, dude. Like, you are culpable for that misinterpretations.

0:20:52 Thomas: Seems to be what they're saying, right?

0:21:04 Lindsey: This was incredibly unclear in the way that they were reporting it. Anyway, so Knox and Momolo, in their response, they said that, you know, in order to make the claim that you're making, you need to have the number of encounters between officers and civilians of various races. In other words, you needs to compute a proportion, and they just don't have that. So this is one of the central critiques here that motivated the authors to retract the article, right? After they were sort of hammered on this for a while.

0:21:04 Thomas: I don't know if we've mentioned this, they retracted the article, it seems, seems like a big deal to me and side note, yeah, it's been cited after being retracted still, right? I mean, people are still referencing it as though it's gospel. Okay.

0:21:30 Lindsey: It is a big deal. It is, it is a huge deal. That. Oh, I'm sure. Yeah. I mean, I can't remember Sam Harris's second second episode when he had his guest on to talk about this stuff. I can't remember when that was released. But this this paper was retracted in I think, of this year. Yeah, it could have been Yeah. Anyway, but yeah, I mean, once you know about the research, if I've already read the article, I'm not going to necessarily know that it was retracted.

0:21:46 Thomas: Okay.

0:22:00 Thomas: That one, so so maybe it would have been after the first episode but before the second one, but but anyway, yeah. Gotcha.

0:22:13 Lindsey: Like you would only know if you, you downloaded the article now, because it says retracted along the top, but all right, so, so that's the first critique right there, they're making a claim here they can't possibly make based on the data, all right, so the second critique is about, like, the specific way that they went about their analyses, um, so the, yeah, we're going to talk about regression for a minute, so in a regression analysis you're, again, you're trying to predict, um, some outcome variable, and and they used a specific form of regression analysis that allows you to predict the outcome of a binary variable. So in this case, they were looking at race as their binary variable, so the person who was shot, were they black or were they white? And then they had other analyses where they were saying, are they Hispanic or are they white?

0:22:18 Thomas: Oh, Okay, yeah.

0:22:56 Lindsey: Yeah. And so what they did was create these prediction models that would allow them to predict the race of the person who was shot on the basis of all these characteristics. So again, they had the age, race, and experience of the officers involved in the shooting. They had shooting characteristics. They had in their data set whether the civilian was armed, whether there was evidence of mental health issues recorded by the police, whether they were reported to have attacked the police officer, and so on. Then they also had information on county characteristics. What proportion of the county was white, black, and Hispanic? What proportion of the homicides committed in the county were committed by white, black, and Hispanic offenders? And so on, right? That actually ends up being a central variable in their conclusions, even though they don't really play that up in the title or the intro.

0:23:47 Lindsey: So that's because again, they're going to make the case that black people get shot because they're criminals. So according to them, right, their results indicate that in fatal officer-involved shootings, the probability that the fatally shot civilian was a minority member versus white did not increase as a function of how many white officers were involved in the shooting. And this is the primary thing that they're using to claim that, like, you know, sort of indirectly, they're using this to claim to, like, so it can't be individual racism on the part of officers.

0:24:19 Lindsey: Because if it was, then more white officers should mean higher probability that the person being shot is a minority, um,

0:24:19 Thomas: Which There seems to Be so much wrong with that, I Of the, this goes in line with one of counter arguments you hear all the time, which is that actually, you know, black officers and Hispanic officers shoot people, you know. Black people and Hispanic people just as much or more. Well, I mean, I think that that predates this paper. I mean that goes back a long way. And you know I've been hearing that argument for a long time. And again, there's two problems with that. One is, again, that individual racism is not necessarily even what we're looking at here. That's not the biggest problem. Maybe a problem, by

0:24:27 Lindsey: Yes. There is.

0:24:42 Lindsey: From This Paper That's Form This Paper Absolutely Right.

0:25:04 Thomas: The way, but not like the biggest issue. It's systemic, not about any one individual bad apple. And two, it assumes that black or Hispanic officers couldn't display racism against black or Hispanic suspects.

0:25:10 Lindsey: Totally. Yes, and I mean, I know that, you know, Sam Harris probably doesn't like the uh the implicit bias literature either, but there, there is evidence from the psychological literature indicating that, like, like, all people, regardless of their race, seem to have these same implicit associations with certain social categories, right? So regardless of who you are, you have an association between young Black males and criminality.

0:25:40 Thomas: Yeah, right, right.

0:25:46 Lindsey: So yeah, no, totally agreed. A lot wrong with this. Especially making it the focal analysis. It's just stupid, stupid. Okay, so additionally, they purported to demonstrate that racial disparities in violent crime rates predicted the race of civilians killed in these fatal officer-involved shootings. So as the percentages, as the percentage of homicides committed by black or Hispanic civilians increased in a county, so did the likelihood of the victims being shot being black or Hispanic. Fun. And then finally, third central claim that was widely reported, in a regression model that purported to represent the typical shooting, and when I hear typical shooting I think, oh, so like the most common shooting, most shootings, a person fatally shot by police was 6.67 times less likely to be black than white, so I previewed this earlier, right?

So now we've got all of our provocative claims covered, right, so claim one, no evidence of anti-minority bias in these shootings, if anything there may be an anti-white bias, two, no evidence that people shot by white officers are more likely to be minorities, and then three, we've got this purported evidence that violent crime rates by minorities is actually what predicts the likelihood of of the person being shot being a minority. So, so those are their, their three central claims, right?

These have been trotted out by a lot of people, um, yeah, so just to recap, so far they looked at this super narrow claim, claim, can we predict whether a fatal shooting victim is white or a racial minority if we have information about the race of the officers involved and the crime

0:26:29 Thomas: Yeah,

0:27:16 Lindsey: Rates of racial minority groups, which doesn't actually have a lot of informational value here, given that they don't have any information.

0:27:21 Thomas: I wouldn't like, me, SJW black lives matter supporter to the stars, would say, if you ask me, like, well, do you think we could predict the race of a shooting victim based on the race of the cop, I'd be like, no, I, I don't think that has anything to do with any of the claims, seriously, am I, am I crazy? That's not at all the issue that people are, are marching about, I don't think.

0:27:43 Lindsey: No, you're not crazy. No. It's not, right? So I'm going to come back to the regression things in a minute because I think that the term prediction is confusing here, but it was designed to be, right? Like they presented their results in this very confusing way, it was begging to be misinterpreted, right? Um, and and then they conclude by implying strongly that, like, well, based on these analyses, the real problem isn't racism, like, it's, yeah, there's... there's layers of uh of problems here, so so let's, let's come back to the the regression thing that I've been teasing so much. So, so second important critique of how these results were misleading is from Schimack and Carlson. This was another published response and reanalysis of the data in this article.

And this response is titled, Young, Unarmed, Non-Suicidal Male Victims of Fatal Use of Force Are 13 Times More Likely to Be Black Than White, which seems to contradict the claims of the Johnson paper, right? Right. And they start with this like really nice, clear explanation of what's actually being tested when you use these logistic regression models. Right. So what are you doing? What are you trying to learn when you set up these models to try to predict the race of these victims? Right. And their ultimate point, where we're going with this, is that the assumptions that you build into your regression models really matter in terms of the ultimate conclusions that you draw from from your results in these analyses.

So when you do regression, you're essentially making point predictions based on various subsets of cases. So if I want to know whether the age of a victim in a fatal officer-involved shooting predicts the likelihood that that person is black or white, the way that I would do that is to predict, based on the observed cases in my set, the probability of being black versus white for like each of the different ages in my set, right?

0:27:58 Thomas: Okay. Yeah.

0:29:39 Lindsey: So if you got a significant relationship there, essentially what that would mean is that, you know, people who are younger, right, are more likely to be black victims of shootings. Does that make sense? Are more likely be black than white within this case? Yeah.

0:29:49 Thomas: Right. Okay, this is a hypothetical, but yeah, okay.

0:29:54 Lindsey: Well, it's kind of is, but like they use, this is one of their variables, right? This is one of the things that they're looking at.

0:29:58 Thomas: Oh, wait, is there actually a relationship between, like, younger... Hmm, interesting. Yeah, I'm trying to think why that would be.

0:30:04 Lindsey: Well that's where we're going. Yeah, so if I was doing this analysis, I, you know, I can use that regression line to estimate how likely I am if I'm a victim of one of these shootings to be black versus white at age 20 versus 35 versus 50 and so on, right? Using my observations. When you use multiple predictors in your regression analysis, and that's what they did in this study, like some of their models had like 15 predictors and you're essentially just like expanding the way that you define that line.

So instead of just predicting the probability of being black versus whites with age, you're also predicting it with the black murder rate in the the county and officer rate and so on, right? So I'm setting all of this up thoroughly in order to say this. They created a regression model in the original paper, the Johnson paper, that they described as predicting the probability of black versus white victims in the typical shooting. And what that actually means is that they're plugging in like an average or middle value for each one of their predictors in order to come up with a predicted probability of whether that the victim was white or black. Does that make sense?

0:30:06 Thomas: Yeah? Okay.

0:31:12 Thomas: I'm not entirely sure I understand that.

0:31:14 Lindsey: Okay. All right. So I'm going to, I'm gonna describe the, uh, the typical shooting model and then see if that, if it does make more sense. Um, right. So, so again, claim from the original paper that we're responding to here is that there was, um, no evidence of anti-black or anti-Hispanic disparities across all fatal shootings. And this was based on the results of regression model of the typical shootings, which suggested that a person fatally shot by police was 6.67 times less likely to be black than white. So what was not obvious in the original paper about that specific claim, that specific analysis, and what Schimack and Carlson made explicit in their response, were the specific characteristics that were being specified in order to predict that value at that, in that quote unquote typical shooting, right? So the typical shooting had the following characteristics. They were specifying a county that had four times more white than black citizens, citizens. So 50 white, 12 black, right.

0:31:20 Thomas: Okay. Huh.

0:32:06 Lindsey: That is typical in in the sense that it's probably representative of a lot of the country, right, but also like the probability of being a white victim of a shooting is likely to go up proportionally with

0:32:13 Thomas: Yeah... Yeah. Well, I got a question. Why do you have to use a typical shooting? Don't you have all the data, can't you just use all, maybe I'm not a statistician here.

0:32:20 Lindsey: No they they did, they did more models, right, um, they, yeah, it's going to take more time than it's worth to get into like each one of the models that they set up, but but this was a, this was a claim that they highlighted, and then they made the claim that like even when we played around with all these variables and the values of all these variables, it didn't change the ultimate conclusion. We still found that white people were more likely to get shot than black people, so typical was like base rate, but then they played around other things, and they were like, no, it didn't change the results, you

0:32:38 Thomas: Okay. Okay, okay.

0:33:00 Thomas: I still don't totally get why you have to use a quote-unquote typical shooting. Like, I feel like if you have the data set, don't you just use the data set and and then try to see what matches up, like what

0:33:13 Lindsey: You do, I think like what you're trying to do in regression is like of get a a summary statistic here of like whether there's a, there's an effect overall. And so I think that that's the point sort of saying like in the typical shooting, right. In the, in our database, right. What I think that their, their point is like, what typically is true is that white people are more likely to get shot.

0:33:35 Thomas: When you talk, but okay, do they have any information on whether or not a typical shooting is something that's actually likely to happen, or could their data be, here's a bunch of shootings from overwhelmingly black areas and here's about shootings from maybe overwhelmingly white areas because we are such a segregated society, and then they like average that out to be a quote-unquote quote, typical shooting.

0:33:36 Lindsey: Hmm. Oh, sorry, I misunderstood. That is exactly the point that we're making. Yeah, at least I think so. So, uh, yeah, let me tell you about all these characteristics, and then yes, that's exactly the point that I want to make is that this typical shooting is not typical. Yeah.

0:34:09 Thomas: Okay. Okay, gotcha. All right.

0:34:17 Lindsey: All right, so in the in the typical shooting, right, quote unquote, these are the characteristics that I had. So again, county had um four times more white than black citizens, and that is important for a reason I'll come back to. Um, the county had uh equivalent rates of homicide perpetrated by white, black, and Hispanic citizens. The average age that was plugged into this regression model was 37 years old. And it was specified that white and black victims were equally likely to have mental health problems, be suicidal, be armed, or attack an officer. And so the odds ratio that was estimated for this group, quote unquote, typical group, for being black versus white was an estimate for that group of shootings, which I agree is probably not typical.

And there's a lot of assumptions being smuggled in there, right? Like the higher number of white citizens, which will inflate the likelihood there. Yeah, it'll inflate the estimate that you have of how likely someone is to be white versus black, right, just as a function of population. The older age, right is important here, because we know that young black Black men are most stereotyped as being violent criminals, they were portraying this in the results as though, you know, this result for the typical shooting meant that in most cases, whites are more likely to be shot than blacks. That's how they were sort of pitching this conclusion. Usually the way that you would do this is create an average point based on the characteristics in your data. And

0:35:03 Thomas: Yeah. Yeah, yeah. Yeah.

0:35:49 Lindsey: Remember, this is like all of the shootings for the United States, right? So this should be representative of all the shootings in the United States, at least in 2015.

0:35:52 Thomas: Mm hmm. Yeah. Did they only use one year? Here?

0:36:00 Lindsey: Yeah, they did. I mean, this is a labor-intensive data set. So to do this for a larger number of cases would be, But yeah, so yeah, usually what you would do in regression is you would use the characteristics of your data set, you sort of, you center your variables on the mean, right?

0:36:01 Thomas: Huh. Okay. Gotcha.

0:36:17 Thomas: Okay?

0:36:17 Lindsey: Right. And then as you're controlling for things, right, you would set certain things to be at the mean or at a middle value, depending on what kind of variable you're using. So I don't, I, yeah, I Think, yeah. And it's not necessarily improper to do that, by and large, but for the specific claims that they're making, right, they need to be super clear about what this means. And just saying that white people are six times more likely to be the victims in the shooting data set than black people in typical shootings vastly distorts what they're actually showing in this regression model,

0:36:25 Thomas: And is that what they did? Okay.

0:36:56 Thomas: Hmm.

0:36:56 Lindsey: Right? So to sort of underline their critique, Schemack and Carlson, after laying all this out, created a second model where they re-specified the variables. Variables, Yeah, um, and this allowed them to generate odds ratios for being black versus white in these incidents for a different subset of shootings. Um, so in this case, they were predicting the likelihood of being black vs. white for shootings in which the civilian is young, age 20, the civilian is unarmed, the civilian showed no signs of mental health problems, the civilian was not uh, there was no evidence the civilian was trying to commit suicide by cop, that was another other thing in their data set and in which the county had an equal proportion of black and white citizens, okay?

And that, again, that's important because it sort of holds the population as something that's not going to be affecting the Prediction of the rate, right? And of course, in Johnson's original analysis, they didn't do that, which is problematic. All right, so in Schemack and Carlson's estimate, using this new model, they found that the shooting victims were were 13 times more likely to be black than white in this other subset of cases. Yeah.

0:37:07 Thomas: Huh. Okay yeah. Wow.

0:38:08 Lindsey: Almost

0:38:08 Thomas: Almost like maybe the other model was misleading.

0:38:13 Lindsey: Like that, yeah. So, so, yeah, so this is this is a second way in which the conclusions of this paper were super misleading. Again, this is one of the claims that was the most widely cited, and It's being really distorted. Sam Harris quoted that specific finding. Specifically, specifically, Did I say specific enough? And yeah, it's wildly misinterpreted. Also, right, I'm done with this study. But also, I downloaded their data set from the publisher's website because they make their data publicly available, and it was nonsense. There were errors in it. There were variables that were mislabeled. I looked at their supplementary Data, yeah.

I looked at their supplemental data sheets, which had, like, descriptive statistics, correlation tables and stuff. Their correlation tables were mislabeled. It looks to me like there were problems in the assumptions for the regression models that they were running that they did not address in the paper. And so, just after looking at this stuff myself, I'm like, this is just sloppy research. This is not... Aside from all the conceptual problems here, they weren't being careful. And if you're doing research on something that has this big of an impact, I can't think of something more irresponsible than to be this sloppy.

0:38:22 Thomas: Yeah. Yeah huh. Really?

0:39:32 Thomas: Yeah, it's a high stakes fake thing. And this is not, I mean, this is not like, okay, they're doing this for Fox News or One American Network. Like, this istrying to be actual science, right?

0:39:34 Lindsey: Yeah. I mean, yeah,

0:39:44 Thomas: Maybe that's why it got retracted.

0:39:43 Lindsey: I Think so. I think that, I means, maybe they felt some trepidation in the wake of this. I hope so. Yeah, okay, so second study, um, this this has an overlapping author list. So this was Cesario, Johnson, and Terrell. This was published in 2019. So two of those authors were on the other papers as well. And the title was, "Is There Evidence of Racial Disparity in Police Use of Deadly Force? An Analysis of Officer-Involved Fatal Shootings in 2015 and 2016." So they had really similar methods. They were actually looking at a similar data set. So it was those cases that were pulled by the Washington Post and The Guardian and things. But they looked at it across two years, 2015 and 2016, um, and they were, they were trying to answer a very similar question, but they did a few things differently, um, so they were looking at, um, they were again using crime rates to benchmark here. So they're trying to predict the race of victims of fatal officer-involved shootings with crime rate?

0:40:46 Thomas: Yeah, wait, did we I don't even think we got to the point in the last paper where you, because like the whole point of the last paper was they don't have on police interactions. How did they estimate that in the past? In newspaper. I don't know if we got that.

0:41:02 Lindsey: It's a great question. No, they didn't. So they are sort of sneakily using the crime rates as a proxy for that. They're saying that, yes, they're saying if the crime rates are higher, then that's a good proxy for officer-involved shootings. I mean, for, excuse me, for interactions with police officers, and that you can therefore use that to benchmark the fatalities in these shootings.

0:41:10 Thomas: Crime rates. Okay. Yeah. I know, and apologies, we tried to get to the second paper, but now I'm realizing there's quite a lot of questions I had from the first one that we didn't get to, which is if you say, okay, let's compare, you know, white people being shot versus black people being shot. And then you say, well, I bet there are more interactions, setting aside the fact of the matter of the interactions, because I think that's a different thing. But if you don't know that data and you say, well, let's use murder rate as a proxy, I don't really think that has much at all to do with interactions.

0:41:32 Lindsey: Mm-hmm. Yep. Mm-hm. Nope.

0:42:00 Thomas: I mean, in a typical murder, you know, like in every, okay, I'll say this, in anecdotally again, but I think this is part of the day that we're talking about, in every shooting, police shooting that I've seen that's a viral video thingy, none of those had anything to to do with murder. I don't know that a one of them had anything to do with, well, this guy's a suspect in a murder. There's traffic stops. There're selling loose cigarettes. There might've used a counterfeit, uh, 20. I can't recall a single one. Maybe there's one somewhere. I cannot recall a one that had anything to do murder rate. And so when, and this was specifically responding, I think to Sam Harris, he used murder rate as a proxy for like, well if they're responsible for this percentage of the murders, then there must be, it's like, okay, setting aside the fact that I do think there is more interaction with police the black community, and that's a problem, a separate problem, you can't really use murder rate to have anything to do with that, I don't think, can you?

0:42:46 Lindsey: Yep. No, No, no. You're exactly right. So here's how they make the case for that. And this is true in both papers. So we are still talking about a second paper. That's good. So the way that they make the case for this is to say that the vast majority of fatal officer encounters happen in the context of them responding to violent crimes.

0:43:27 Lindsey: And they make the claim that, well, it's like 5% or less that die outside that context. And so this has to be a pretty good proxy, right? No, they're wrong. They're wrong and you're right. Yeah. Yeah. Yeah, they're definitely wrong and you're definitely right. Um,

0:43:11 Thomas: Okay, hmm, okay. Oh, they are wrong? Oh, how do you know that? How do you, you know, I'm right, I was just guessing, okay, you're a careful sciencey person, so I assume you're not just saying that based on nothing, hmm?

0:43:44 Lindsey: But that's, I'll tell you, I'll telling you right now. No, no, I'm not. Yeah. So, so here's, here's the, here is the thing, right? Um, and I'll, I will go ahead and I will talk about the, the Ryan and Evers response here. Um, so, Ryan and Ever responded to the second paper, right, and again, they are doing very, very similar things trying to predict race here. They break things up by, again, like they use crime rate as a proxy to predict the race. And they're also, they do one other additional thing from the first study, which is that they look at these analyses separately for like different types of police encounters. So they have like one analysis they do is just like all encounters in which people were fatally shot. Then they have a second analysis where they look at all encounters in which police were fatally shot when the person is unarmed and not aggressing.

0:44:40 Thomas: Right.

0:44:40 Lindsey: And then they have a third category that is all encounters in which the person was shot, and there was, and the reason that they were shot is because the police mistook an object that they were holding for a weapon. And yeah, yeah, yeah. So in each of this speaks exactly to the point you were just making, right? For all of those three categories of shootings, though, though. They benchmark it on the violent crime rates. How does that make any sense?

0:44:52 Thomas: Which is, yeah, according to them, yeah, yeah, that doesn't make sense, yeah.

0:45:11 Lindsey: It doesn't make sense. So Ryan and Evers published a response to this, and they make this claim, you know, just purely on logical grounds that, again, Cesario's data were insufficient to test their central claims here. So they say, right, that they benchmarked, so Cesario and all benchmarked all fatal officer-involved shootings using only police encounters regarding violent, or using only violent crime rates, I should say, right? And Ryan and Evers argued that instead, right, in order to do a proper comparison, they either should have compared all fatal officer-involved shooting to all police encounters, full stop, or fatal officer involved shootings in the perpetration of violent crimes to police encounters regarding violent crimes. Um, yeah, and just, just to, exactly,

0:45:53 Thomas: Separately, yeah, yeah, Yeah, okay, not blending them all together and comparing it to violent crime or something like that, gotcha. Yeah, that

0:46:07 Lindsey: Right, and they they they Use yeah, it does, and they use an analogy that I really like to sort of just show how this is, a this is a problem with the math, right? So they were like, if you want to determine whether there's a gender bias in NSF winners in engineering, you should compare the gender ratio of NSF winners in engineering to the gender ratio of applicants within engineering, right?

What you should absolutely not do, I'm still quoting, is compare the gender ratio of NSF winners overall to the gender ratio of applicants only within engineering. Which, just to illustrate, would show a massive anti-male bias, right. Because in engineering, you've got more men than women, so you'd think that there's going to be more male applicants than women. But if you look at winners overall, overall, that's probably going to be closer to 50-50 in terms of men and women winning, because women are better represented in other disciplines that would get NSF awards, right? So if you make that inappropriate comparison, then it's going to look like, hey, more men than women are applying, but equally men and woman are getting it. Men are discriminated against. So they continue, quote, this incorrect approach is the method that Cesario uses in his paper. He compares all of those who shot, who are shot by police, to only violent criminals, and so Cesario's

0:46:09 Thomas: Makes sense. Okay, right, yeah, right,

0:47:26 Lindsey: Method could only be correct if and only if every person shot by police was a violent criminal, which we know is not true.

0:47:32 Thomas: Right, Right. Right, yeah, that's the point of making, okay, I like these people. Good response, out of 10.

0:47:33 Lindsey: Uh, yeah, It was a wonderful response yeah, And so I know, I know we're out of time, but I'm gonna, I mean, I'm going to give you all these papers, yeah,

0:47:44 Thomas: No, it's yeah, take, take a minute here, finish off the second paper, wow.

0:47:48 Lindsey: So, um, so yeah, I'm going to give you all these papers, but the response that we definitely don't have time to get to essentially takes that argument, that response that we just talked about, and then they prove it mathematically. They come up with a better analytical strategy. They demonstrate that in a hypothetical world in which there really is anti-Black bias, right? Obviously, it's not Earth, but maybe it's Mars, right, in a hypothetical world in which there absolutely is anti-black bias, the strategy that these original authors use, that Cesario used to analyze their data, would not show it, and actually would show an anti-white bias, which is what they claim to show. And then it, yeah,

0:48:03 Thomas: Wow. Wow. So, they're saying it's kind of poison from the beginning, there's no, like, even if even if they were right about their claims, they're, there, the way they modeled it and showed us the results wouldn't even show us that, okay, do,

0:48:38 Lindsey: Wouldn't show it. Wouldn't show it. And, and then on top of that, they go back to the original data set, they reanalyze it with a different benchmarking method that, yeah they do. So they use a different benchmarking method that they have already, you know, sort of showed mathematical proofs actually would yield an unbiased estimate. And they find, surprise, surprise. This is important. They find in their, in their reanalysis, um, that there does seem to be disproportionately more black people killed in the types of shootings where they are unarmed and not aggressing, but there is equivalency. There doesn't seem to Be racial disparity in the category where the person is is aggressing and armed.

0:48:47 Thomas: Their work for them, right, yeah, okay, yeah,

0:49:22 Lindsey: So no racial bias there. And the reason I think this is important is that the fact that you don't get racial bias in unambiguous situations, but you do get it in ambiguous situations, yeah, it's exactly consistent with what we know from the psychological literature about how bias operates interpersonally. When it's unambiguous, this person is attacking me, it doesn't matter whether they're black or white, right? But if you have an ambiguous situation, and just to talk about a different situation, like if you're evaluating a black or a white candidate for a job, and the black person is clearly more qualified than the white person, they're going to get hired. But if it's ambiguous as to which one is better, that's where bias tends to operate if it happens interpersonally.

0:49:34 Thomas: That kind of makes sense, mm-hmm.

0:50:09 Lindsey: Right? Yeah. So if if it's ambiguous, totally, totally.

0:50:10 Thomas: And by the way, all of this data though is still according to police reports, isn't it, or it's still like self-reported, you know, so it's, it's, take all of it with a grain of salt, because the, even the ambiguous, or you know, the unambiguous, who knows what really happened there, and

0:50:28 Lindsey: Yes. No, you're absolutely right about that. And that's a super important point to make. They are focusing on one tiny piece of it. They're doing their math wrong. They're being sloppy. And it's bad.

0:50:41 Thomas: So, was, sorry, is this all still the first paper, or were you debunking this is the second paper, you're debunked, oh, okay, so this one, so when's this gonna get retracted? Gosh, well, it's the, some of the same people, right? They should know how to retract it, yeah.

0:50:43 Lindsey: Being ignored. Nope, that's, that's Yeah, we'll see. We'll see Yeah, they they should, they got practice now.

0:50:59 Thomas: They don't know which forms to fill out, or whatever the heck they have have to do? Which website to go to?

0:51:06 Lindsey: Yeah, so no, I mean, I think that that's a good point to go out on, you know, like, there are so many, there's so many other places at which there is bias in policing. And we have lots of good data on that. We know that black people get stopped more often, we know that they get searched more often. We know that there are all of these ways in which there is biased operating on every level. And again, these, these researchers are just sort of like pushing all that to the side and being like, yeah, but you know how many cops are white supremacists, that's not appropriate, so mm-hmm.

0:51:34 Thomas: Yeah. Yeah. I, I have so much yelling to do. I'm telling you, I've got so much Yelling to do about this. We're going to do it on a, on a bonus episode where we yell at Sam Harris and other people making these terrible arguments in a moral, morally superior way. When in fact they're actually cherry picking the data and they're not on solid footing. It is, it's infuriating, absolutely infuriating.

0:51:39 Lindsey: Sweet.

0:51:58 Lindsey: It is agreed.

0:51:59 Thomas: So we've got yelling to do. That was wow. I I think you did a great job, Lindsay, in explaining it without the use of it. You don't get any visual aids or anything. You don't get a laser pointer.

0:52:00 Lindsey: Yeah.

0:52:10 Lindsey: I know, I thought about making PowerPoints, but, you now, yeah.

0:52:14 Thomas: You you doing your best to help us non-science people, us science noobs, understand the trickery that's going on here. That's so helpful. That is an excellent breakdown. Hey, if you see these claims being made somewhere, please share this. It is maddening. It's everywhere. I am just surprised. I have not heard about this before, a good response to this. Now, it sounds like people are issuing studies that reply to these other studies or whatever, papers that reply part of these papers. So that's great. You could also link that, but help get the word out that this is crap, that these studies are not, you know, that this paper is one of which has been retracted. The other is not on solid footing and maybe hopefully, in my opinion, should be retracted, soon to be retracted how infuriating, but excellent breakdown, Lindsay. Thank you so so much. And, um, you know, let's do some yelling on a patron only episode.

0:52:43 Lindsey: Yeah. But those don't go viral, oh,

0:53:20 Clip: Thank you so much for listening to Serious Inquiries Only. If you like the show, the absolute best thing you can do is support us at patreon.com/seriouspod. The second best thing you can do, is please share it on social media or by word of mouth. Speaking of social media, follow me on Twitter at Serious Pod. If would like to be Facebook friends, send me a request at facebook.com Thomas podcast. If not up for that level of intimacy, that's fine. Maybe you want to join the discussion at facebook.com/groups/SIO members. Thank you again for listening. We'll see you next time.