Readers requested us to fact-check Obama’s broad statement at the Dec. 1 Paris news conference, that “mass shootings just [don’t] happen in other countries.” Critics pushed back on that comment immediately, noting that Paris – where Obama was speaking – had just experienced a mass shooting. Mass shootings do happen in other countries, and that statement clearly is false.

Then, in response to the Dec. 2, 2015, mass shootings in San Bernardino, Calif., Obama used more specific language that clarified what he was referring to. Indeed, he has made versions of this claim after other recent mass shootings. So we explored the relevant data and definitions, what exactly Obama is referring to, and the caveats associated with comparing mass shootings across countries.

Here’s an important caveat to establish: There is no consistent definition of “mass shooting” or “mass public shooting” across countries, or even among researchers who track them within the United States.

The FBI does not officially define “mass shooting” and does not use the term in Uniform Crime Report records. In the 1980s, the FBI established a definition for “mass murder” as “four or more victims slain, in one event, in one location,” and the offender is not included in the victim count if the shooter committed suicide or was killed in a justifiable homicide, according to a Congressional Research Service report detailing the definitions.

After the 2012 shootings in Newtown, Conn., Congress defined “mass killings” to mean “three or more killings in a single incident.” Some media outlets and researchers still use the four-fatality definition, and have adopted the CRS definitions of “mass shooting” and “mass public shooting.” Other researchers include injuries in the victim count. Some researchers include acts of terrorism, drug deals gone wrong or gang conflict in their researchers. Others don’t.

Some media reports and advocates use a broader definition used by the Mass Shooting Tracker maintained via Reddit, an online forum. In this case, mass shootings are incidents in which four or more people, including the gunman, are killed or injured by gunfire. By this count, the San Bernardino shooting is the 355th mass shooting this year. (In comparison, CRS counted 317 mass shooting incidents from 1999 to 2013.)

An FBI spokesman said such counts of shootings to include injuries would be categorized as a “mass casualty” event.

While Obama incorrectly said during the Paris news conference that mass shootings “just [don’t] happen in other countries,” he often has clarified that he is referring to the “frequency” of shootings in the United States compared to advanced countries. And his use of “frequency” appears to be the actual count of shootings in the United States.

White House officials did not say what Obama means by “frequency,” but they sent several news sources to support his claims. They noted how the U.S. has more gun violence, in general, than other countries, as reported by The Washington Post and Vox. (The Fact Checker has examined Obama’s rhetoric on gun homicides in the past.)

The White House also pointed to research by University of Alabama criminal justice professor Adam Lankford, who declared mass shootings the “dark side of American exceptionalism.” The paper is not yet published officially, but his findings have been covered widely in the news and has been used to support Obama’s argument.

Lankford ran statistical analyses of the total number of public mass shooters per country from 1966 to 2012 in 171 countries, and controlled for the national population size.

He found that the United States had far more mass shooters (90 shooters in the 46 years) than the other countries, which averaged 1.7 public mass shooter per country. His research excluded gang-related shootings, drive-by shootings, hostage-taking incidents, robberies, and acts of genocide or terrorism. (Lankford requested we not distribute his unpublished study and declined to provide the underlying data, citing ongoing research.)

Lankford said he looked at the actual count of shooters rather than the per capita rate of incidents because mass shootings are rare events, and small populations of other countries can inflate the rate. He said looking at rates of incidents are “wildly misleading” – for example, due to the Umpqua college shooting in Roseburg, Ore., the city’s public mass shooter rate (number of offenders per capita) would be higher than most American cities because of that attack. But the rate would reflect Roseburg’s tiny population, and not necessarily mean that Roseburg is at higher risk in the future, he said.

Rates need to be interpreted with caution, Lankford said. One rate that he calculated was between the United States and the European Union, “because the populations are so large for each that the rates become more reliable. The number of public mass shooters per capita (the “rate”) for the U.S. was approximately 5 times the per capita rate for the European Union.”

Other researchers disagree.

State University of New York-Oswego public justice professor Jaclyn Schildkraut and Texas State University researcher H. Jaymi Elsass have been tracking mass shooting incidents in 14 countries from 2000 to 2014. They compared the United States to 11 other countries (Canada, Finland, China, England, Australia, France, Germany, Mexico, Norway and Switzerland), and found the United States had a lower rate of mass shooting fatalities per 100,000 people than Norway, Finland and Switzerland.

Other than China, these countries were all member countries of the Organization for Economic Cooperation and Development, the common measure for “advanced” countries. But the actual count of incidents showed the United States had 133 shootings during the period, compared to maximum of six in each of the other countries.

In their comparison of the United States to 19 other, mostly non-OECD countries, Lebanon rated higher than the United States in mass shooting fatality rate. They excluded shootings related to gang violence or targeted militant or terrorist activity.

There are caveats to these data. The researchers also looked at mass shootings that resulted in multiple injuries, not just fatalities. This could have driven the number of shootings up, especially for the United States. For comparison, Mother Jones’s mass shootings database of four or more fatalities from 1982 to October 2015 lists 72 shootings.

Plus, the Schildkraut/Elsass list is not exhaustive; they are still compiling the list of shootings in other countries, and it does not include all of the shootings that may fit their definition.

It’s also important to note that Norway, Finland and Switzerland all had one or two incidents each that left multiple dead or injured. The United States, in contrast, had 133 shootings that killed or injured multiple victims, according to their research.

For example, a single 2011 attack in Norway, a country of about 5 million people, killed at least 67 people. On a per capita basis, that equates to about 5,000 victims in the United States. In contrast, there were at least four mass shootings that killed four or more victims in the United States in 2011, but it did not add up to the number of people who died in Norway, Schildkraut said. (PolitiFact used this research to rate Obama’s claims from June “Mostly False.” The White House sent us an article disputing that rating.)

Schildkraut said it’s “absolutely not fair” to count the sheer number of incidents of shootings, which shows the United States ranks far higher than any other country. Plus, comparing shootings across countries is an apples-to-oranges comparison, because of gun policies, politics and attitudes are unique to each country, she said.

John R. Lott Jr., gun rights analyst who has tracked mass shooting rates in the United States and European countries, said Obama’s references to “frequency” are problematic and inaccurate: “If you are going to compare the U.S. to someplace else, if you are going to compare it to small countries, you have to adjust for population. Alternatively, compare the U.S. to Europe as a whole.” Comparing to the U.S. to Europe (including OECD and non-OCED countries) from 2009 to 2015 shows the rate of mass shootings in the U.S. and Europe are about the same, Lott said. (Lott uses the FBI definition of four or more killings in a public space, excludes gang or crime related activity, and includes acts identified as terrorism.)

Astute readers might notice how Lankford and Lott both compared the United States to grouped European countries, but their conclusions are vastly different. Lott says the rate is about the same, while Lankford says the rate is five times higher in the United States. How is this possible? The researchers are looking at different sets of years and different sets of countries (Lott looked at Europe as a whole; Lankford at European Union). Lott uses a broader measure of mass shootings than Lankford does. Lankford looks at number of shooters, Lott uses fatalities and shooting incidents. This is an example of how the data and definition can be adjusted to show different findings about mass shootings, even using per capita rate.

The most accurate way that Obama has described shootings, Lott said, was his statement after the Oregon shooting: “We are the only advanced country on Earth that sees these kinds of mass shootings every few months.”

Still, Lott added: “I don’t know what sense it makes to say, ‘I have an area of 320 million people, and I’m going to compare it to 8 million, or even 40 million people. I would expect, just out of randomness, to have more of whatever event to happen in a country of 320 million people.”

But John Roman, senior fellow in the Justice Policy Center at the Urban Institute, agreed with the underlying message of Obama’s statement, regarding the prevalence of mass shooting incidents in the United States. Frequency is “about how often something happens, not about how many people were affected by any single event,” such as the 2011 Norway shootings, he said.

“Yes, it does happen in other places. But boy, does it happen a lot in the U.S., and boy, does it happen really frequently,” Roman said. “And it happens without cause, without reason, without some ideological backbone.”

Why it’s hard to figure out

Whenever a mass shooting occurs, a flurry of infographics floods social media, a range of facts are cited and rhetoric swirls amid developing news. It doesn’t help that Obama uses inconsistent and sometimes vague language to describe mass shootings in the United States compared to other countries.

At times, his description was wholly misleading and inaccurate (mass shootings “just [don’t] happen in other countries”); other times, his description was quite accurate (“We are the only advanced country on Earth that sees these kinds of mass shootings every few months”); and other times, it seemed to be somewhere in the middle (the “pattern now of mass shooting that has no parallel anywhere else in the world”).

Quantitative measures of cross-comparative crime statistics, especially where the crime is not consistently defined (i.e., “mass shooting”), usually ends up being apples-to-oranges comparisons. (We dug deeply into cross-comparative measures of the U.S. criminal justice system to other countries in the world, and it wasn’t easy. )

It’s not just about population size, but also about differences in gun culture, policies and politics in each country. How can one compare Israel (where there are government-issued guns) to China (with stringent gun laws) to the United States (with Second Amendment rights)?

We are sensitive to the tragedy of each mass shooting. Our goal is to provide the underlying data and definitions – which show there are numerous ways to talk about mass shootings in the United States, and around the world. We urge the president to be more consistent and precise in describing mass shootings in the country (as he did after the Oregon shooting) rather than using vague or misleading phrases.

“The one thing we do know is that we have a pattern now of mass shootings in this country that has no parallel anywhere else in the world, and there’s some steps we could take, not to eliminate every one of these mass shootings, but to improve the odds that they don’t happen as frequently.”

-President Obama, interview with CBS Evening News with Scott Pelley, Dec. 2, 2015

“With respect to Planned Parenthood, obviously, my heart goes out to the families of those impacted. . . . I say this every time we’ve got one of these mass shootings: This just doesn’t happen in other countries.”

-Obama, news conference at COP21 climate conference in Paris, Dec. 1, 2015

“We are the only advanced country on Earth that sees these kinds of mass shootings every few months.”

-Obama, statement on shootings at Umpqua Community College, Roseburg, Ore., Oct. 1, 2015

“You don’t see murder on this kind of scale, with this kind of frequency, in any other advanced nation on Earth.”

-Obama, speech at U.S. Conference of Mayors, June 19, 2015

“At some point, we as a country will have to reckon with the fact that this type of mass violence does not happen in other advanced countries. It doesn’t happen in other places with this kind of frequency. And it is in our power to do something about it.”

-Obama, statement on the shooting in Charleston, S.C., June 18, 2015

Copy the Story Link