May 22: John Lott Responds

E-mail us your comments (we may post them)

Reason's gun page

John Lott’s 1998 book, More Guns, Less Crime contains many points with which I agree. For example, I believe that many criminals are leery of approaching potential victims who may be armed -- an idea at the core of his deterrence theory that guns help to prevent crime. I also believe that violent criminals are not typical citizens, and that the possession of a gun by a law abiding citizen is unlikely to turn him into a crazed killer. Lott also has a point when he speaks of the over-reporting by the media of gun violence by and against kids and the corresponding under-reporting of the defensive use of guns to prevent crime. As a gun owner myself, I was quite prepared to accept Lott’s thesis that the positive deterrent effect of guns exceeds their harmful effects on society, but as a scientist I have to be guided by what the data actually show, and Lott simply hasn’t made his case. Here’s why:

Lott misrepresents the data. His main argument that guns reduce crime is based on the impact on various violent crime rates of "concealed carry laws," which allow any legal gun owner to carry concealed weapons. Since these laws were passed at different dates in different states, he looks at how the crime rates change at t=0, the date of the law’s passage in each state. Lott’s book displays a series of very impressive looking graphs that show dramatic and in some cases immediate drops in every category of violent crime at time t=0. The impact on robberies is particularly impressive, where a steeply rising robbery rate suddenly turns into a steeply falling rate right at t=0 -- almost like the two sides of a church steeple. As they say, when something looks too good to be true, it probably is. Lott neglects to tell the reader that all his plots are not the actual FBI data (downloadable here), but merely his fits to the data.

The actual data are much more irregular with lots of ups and downs, and they show nothing special happening at time t=0. Lott has used the data from 10 states in his book. When we look at changes in the robbery rate state by state, only two of the states (West Virginia and Georgia) show decreases at t=0, while the other eight show increases. Overall, averaging the 10 states, there is a small but not statistically significant increase in the robbery rate at t=0, certainly not the dramatic decrease Lott’s fits show. In fact, Lott’s method of doing his fits is virtually guaranteed to produce an "interesting" result at time t=0. What he does is fit a smooth curve (actually a parabola) to the data earlier than t=0, and fit a separate curve to the data after t=0.

Given a completely random set of data, Lott’s fitting procedure is virtually guaranteed to yield either a drop or a rise near time t=0. Only if the data just happened to lie on a single parabola on both sides of t=0 would the fits show nothing special at that time. Since random data would show a drop or a rise equally often at t=0, we have a 50 percent chance of finding a drop -- not a very good argument for the drop being real. The fact that all categories of violent crime (murder, rape, assault, robbery) show drops is also not particularly surprising, since the causes of violent crime (whatever they are) probably affect the rates in all the separate categories. Similarly, it is no more mysterious that when the overall stock market rises or falls dramatically the individual sectors (industrials, utilities, etc) are more likely than not to move in the same direction.

Lott’s results are not consistent. Taking Lott’s fits at face value, we find they give inconsistent results. For example, he shows murders, rapes, and robberies each declining sharply and immediately at t=0, the year of passage of the laws, but the aggravated assault rate rises slightly and doesn’t start its descent until three years after the law’s passage. Presumably, the same sorts of folks are committing murders and assaults, so this difference is very puzzling. Similarly, Lott shows the rate of multiple public shootings declining dramatically (by 100 percent) only two years after t=0. But using follow-up data in a more recent paper, Lott shows multiple shootings rising precipitously the year before t=0 and then declining right at t=0. It’s difficult enough  understanding why the impact of the laws should be so much greater on multiple shootings by crazed killers than ordinary murders (which drop only 10 percent), but figuring out how the laws could work in reverse time on the thinking of these psychos is a real challenge.

Lott cannot account for all the relevant variables. Recognizing that violent crime rates can depend on all sorts of factors aside from the passage of concealed carry laws, Lott includes many variables when he runs his multiple linear regressions to disentangle the impact of each factor. Many of these variables, such as arrest rates, percentage of African Americans, and population density, account for a far greater percentage of the variation in violent crime than the mere 1 percent he attributes to passage of the laws. However, with such a small dependence on the one factor he is looking for, only if Lott has included all the relevant variables that could affect the rate of violent crime can he hope to see the residual amount due to the effect of that one factor. In answer to this criticism Lott says OK -- tell me what variable I’ve left out and I’ll include it. But, the list of plausible variables that could affect violent crime rates over time is virtually endless. Here, for example, are 14 that Lott didn’t include: 1) amount of alcohol sold, 2) price of alcohol, 3) amount of drugs sold, 4) price of drugs, 5) number of police on the beat, 6) number of police brutality complaints, 7) average summer temperature, 8) number of convicted felons on the streets, 9) average age of convicted felons on the streets, 10) percentage of teenagers living in two parent households, 11) high school dropout rate, 12) dollars spent on crime prevention programs, 13) minimum wage rate, 14) amount of media violence. I’m sure readers could come up with many more plausible factors, any one of which could mask the true dependence on the concealed carry laws.

Lott doesn’t properly compute statistical significance. Another very serious problem with Lott’s method is how he calculates the statistical significance of his results. He essentially asks, What is the probability of getting the observed variation of the crime rate on either side of t=0 based on changes in the various socio-demographic variables and random variations? If that computed probability is very small, he regards his hypothesis that the concealed carry laws made the difference as being proven. But, that’s not right. He needs to look at the probability of a change in the crime rate for years t= -3,-2,-1,0,1,2,3, etc. Only if the probability is very much less for year zero than the other years can he consider his results meaningful. It seems very likely, however, that Lott would find similarly low probabilities for all these other years, because only if the violent crime rate were static over time would there be no significant variation on either side of year t=0, or any other given year. In fact, one researcher’s analysis of Lott’s data show that the most significant turning point for the robbery rates occurs before t=0.

Lott has correctly observed that by passing concealed carry laws in various states in various years, the U.S. has been in effect conducting an extremely interesting social experiment. That experiment, in principle, can give us an empirical answer to the relationship between easing restrictions on gun-carrying permits and crime. However, his one-sided analysis of the data inspires little confidence that we can count on him to tell us the true results of this experiment. From all indications it seems that the concealed carry laws probably have had almost no effect, one way or the other.

Robert Ehrlich is a  professor of physics at George Mason University and author of the new book, Nine Crazy Ideas in Science: A Few May Even Be True (Princeton University Press)

May 22: John Lott Responds

E-mail us your comments on the gun debate.