Can the news help you learn statistics? In this series of studies, we’re asking people to read, watch, or listen to different versions of a news report that contains numbers, visualizations, or both. Then we’re asking them a series of questions about the credibility of that news report and some of the inferences they make from that presentation. These are the two dependent variables common to all our studies. In addition, we are asking people to assess the relevance of the story topic to four ever-widening social scales: me, my close family and friends, people who live near me, and society as a whole. For details about Numbers in the News and the hypothetical model that underlies this research, click here.

Official Statistics

In 2017, statisticians Iddo Gal and Irena Ograjenšek argued that adults need help making sense of official statistics. Official statistics are “data and diverse information products [made] available to keep policy-makers, various user groups, and the general public apprised of the current economic and social situation” (Gal & Ograjenšek, 2017, p. 86). That includes statistics based on censuses and other government surveys, as well as data like birth rates and tax information that’s collected for primarily administrative purposes (Bulmer, 1980).

As these examples suggest, collecting official statistics makes use of the resources and functions of government. Yet official statistics – like all statistics – should not be confused with precise, complete counts. To consider just one example, the birth rate: not all births are formally registered, and there may be delays in reporting even among those that are.

Furthermore, there is often confusion about what precisely is being counted. In official contexts, unemployment may refer either to the number of people collecting unemployment benefits or to the official Labor Department unemployment rate, which is estimated based on a representative sample of the U.S. population. And there are many people that might be thought of as unemployed who would not be counted in either of these numbers: people who have given up looking for a new job, self-employed people whose work has dried up, people who don’t feel safe working due to COVID-19 but don’t qualify for formal accommodations.

What we saw

As part of our study about estimates, we asked respondents to tell us which of two unemployment statistics they found more meaningful and why. In this post, we’re taking a deeper dive into those open text responses.

The original news story from which the excerpt was taken read as follows:

The coronavirus crisis has sent U.S. unemployment surging to 14.7%, a level last seen when the country was in the throes of the Depression and President Franklin D. Roosevelt was assuring Americans that the only thing to fear was fear itself.

And because of government errors and the particular way the Labor Department measures the job market, the true picture is even worse. By some calculations, the unemployment rate stands at 23.6%, not far from the Depression peak of nearly 25%.

The Labor Department said Friday that 20.5 million jobs vanished in April in the worst monthly loss on record, triggered by the coast-to-coast shutdowns of factories, stores, offices and other businesses.

In the edited version, we replaced “By some calculations” in the second paragraph with “If you count people who are not looking for a new job,” based on information provided later in the same article.

The version people read didn’t have much effect on which number (14.75% or 23.6%) they found more informative. But it did seem to impact some of their reasoning about why.

Not all open-ended responses addressed this question of official statistics. Instead, some respondents focused on the inclusion and exclusion criteria of the two numbers. A number of people – particularly among those who saw the “If you count people who are not looking for a new job” version – thought the unofficial numbers were inflated and thus not trustworthy. These responses described people who are not actively seeking work as “unemployed by choice” or those who “want to be unemployed.” In contrast, some other respondents pointed out that the official number does not include freelancers, gig workers, or those who have given up on looking for work because there are no available jobs. Many of the people who discussed the criteria for inclusion provided additional information not provided in the excerpt.

Of the 121 people who did address official statistics, 33 contrasted “official” numbers with “estimates,” without recognizing that all statistics rely on models and estimation. As one respondent wrote, “That [14.7%] number is officially confirmed, and calculations by the government are done in a certain way that best shows the condition of the economy.” While the second paragraph alludes to “government errors,” these respondents seemed inclined to trust official numbers regardless.

We did not tell respondents the source of the text, and at least 20 people said they were inclined to trust the Labor Department numbers in the absence of a clear sense of who had provided the alternative calculation. That is, many people are – rightly – skeptical of unsourced numbers. *"It seems we should trust the Labor Dept. over whatever this source says ‘the true picture’ is, especially since I don't know the source."* All but two of these respondents saw the “by some calculations” version, and that was frequently reflected in their wording: “I do not believe in some calculations, I believe in what the Labor Department measured.”

On the other hand, a small number of responses also indicated a reflexive mistrust of government statistics. One person wrote that “[23.6% is] not the official number, so is probably more accurate.” and others were even more direct in their criticism, like the person who wrote, “I feel like the government lies a lot of the time and doesn't want people to know the actual numbers in order to supress [sic] or lower panic emotions from a lot of people.” That said, we believe that many of the respondents who suggested that 23.6% was “true” or “real” were responding directly to the text, which contrasts the official number and the “true” one.

Finally, at least 7 respondents correctly acknowledged that official statistics are collected the same way over time to allow for comparison. Even with some of the problems identified in the article, these respondents noted that collecting information the same way over time was still valuable. Some respondents accepted the article’s premise that the higher rate was more true – and still identified comparability as an important consideration: “The officially reported unemployment rate is the number most people see or know about. Not many know about the truer unemployment rate figure of 23.6 unless you have a deep love of economics or business to know that unofficial figure exists.”

Let’s Put it to Work

Official statistics are valuable for understanding societal and economic trends. Like all statistics, they require interpretation. No single number can provide the whole picture, and no statistic is 100% certain. The government’s numbers are subject to the same constraints as all other statistics.

During the COVID-19 pandemic, official data is everywhere: case counts, mortality rates, economic indicators. Both individuals and governments are assessing risk based on these numbers. In order to make informed decisions, it is essential to understand what information the data can tell us – and what it can’t.

Photo by Kim Gorga on Unsplash