Estimating state and local employment in recent disasters—from Hurricane Harvey to the COVID-19 pandemic

The U.S. Bureau of Labor Statistics (BLS) Current Employment Statistics (CES) program produces some of the timeliest economic indicators each month, both for the nation as a whole and for varying levels of geographic detail. The CES program conducts a monthly survey of about 144,000 businesses and government agencies representing about 697,000 individual worksites. Respondents to the CES survey provide information on the number of employees, total payroll, and hours paid for the pay period that includes the 12th day of the month. The CES program uses these data to produce estimates of industry employment, hours, and earnings. BLS typically publishes national data within a week of the end of the reference month; data for the 50 states; Washington, DC; Puerto Rico; the U.S. Virgin Islands; and about 450 metropolitan areas are released approximately 2 weeks later.

Natural disasters challenge assumptions built into the employment estimation process regarding nonresponse and the relationship between business openings and closures, and the CES program must address this issue in order to produce accurate data. This article provides an overview of three aspects of the CES methodology that natural disasters confront. It then provides case studies of how the CES program dealt with these questions in developing state and metropolitan area estimates following five recent hurricanes—Harvey, Irma, and Maria in 2017, and Florence and Michael in 2018—and how the lessons learned from the earlier experience were applied to producing estimates in the early months of the coronavirus disease 2019 (COVID-19) pandemic. (See table 1.)

CES methodology and natural disasters

The CES program surveys businesses each month and uses the data and a consistent, objective methodology to produce estimates of industry employment, hours, and earnings. The methodology relies on assumptions that are continually evaluated and generally hold in normal times. However, natural disasters are extreme events, and this section explores three ways that such disasters raise questions about the usual CES methodological underpinnings.

Are survey respondents different from nonrespondents?

The CES survey is designed as a probability-based survey.¹ The BLS Quarterly Census of Employment and Wages (QCEW), which contains information on all employers required to participate in the unemployment insurance (UI) system and covers about 97 percent of all nonfarm payroll employment, provides both a sampling frame and the main benchmark source for the CES program.² In developing the QCEW as a sample frame, BLS stratifies the data by state, industry, and establishment size. The CES survey randomly selects businesses within these strata, with the inverse of the probability of selection used as the sample weight in estimation. The survey estimates are benchmarked to the administrative QCEW data annually. The difference between the benchmark data and the survey-based estimates—the benchmark revision—is regularly used to assess the accuracy of the CES estimates. After setting benchmark employment levels each year, the CES program estimates monthly employment growth rates with a weighted link-relative estimator. The estimator is essentially the ratio of the current month’s weighted employment to that of the prior month in the set of matched respondents that reported nonzero employment for the pay period containing the 12th of both months. The generally small number of active survey units reporting zero employment in either the current or prior month is not used because of assumptions in the business birth–death model (discussed in the next subsection).

Equation 1 provides the weighted link-relative estimate () for time t, where  is reported employment for a matched respondent i in n matched sample establishments, w is the sampling weight, and d is a nonresponse weight adjustment (calculated for industries identified to respond at substantially higher or lower rates)³:

(1)                      .

Within each estimating domain (an area or industry combination in which the estimate calculation is performed—other published cells are summaries of estimating cells), the estimator implicitly imputes data for nonrespondents as well as businesses reporting zero employment in the current or prior month by using the growth rates of the matched units. State-level estimates are produced for “estimation supersectors,” high-level industries (such as construction or leisure and hospitality) that are estimated directly and control the sum of estimates of more detailed industries. The sample size in these estimation supersectors is large enough to avoid using small-area modeling techniques in most cases, but domain estimates use respondents in businesses from quite different industries (for example, clothing stores, gas stations, and grocery stores mixed together in retail trade).

The implicit imputation follows from the form of the estimator, which does not require explicit imputation, and a missing-at-random assumption within each estimating cell (conditioned on the industries in d_i). The survey respondents stand in for the nonrespondents, and this assumption works well when they are similar, on average.⁴

Natural disasters challenge the missing-at-random assumption. Certain types of businesses—for example, those in federally designated disaster areas—might have more job losses than others in their state because of business disruptions, and they might respond to the CES survey at a lower rate, as other matters take a higher priority. Similarly, certain industries within an estimation supersector may fare differently from one another during a disaster in a way that correlates with response rates, and the same may be true for differences in employment trends and response rates between large and small establishments.

Are business births and deaths properly captured?

The ability to use the QCEW as a frame with a close match to the target population of all nonfarm payroll jobs is one of the CES program’s greatest strengths. Two of the program’s biggest challenges are accounting for businesses that have opened and closed since the frame was established. It is impossible to sample, enroll, and collect data on business births in real time because the QCEW lags the CES survey by several months. Establishment deaths—defined as worksites that no longer have any payroll employees, which may be temporary or permanent—present a challenge because businesses generally stop reporting when there are no employees. A small number of businesses report establishment deaths; however, they are mainly firms with multiple worksites. These two components of job growth generally offset one another, and the degree to which CES procedures do not capture the residual growth when applied to historical QCEW population data is known as the “net birth–death residual.” This value is generally small and stable, and therefore the CES program forecasts historical values with a time-series model for use in estimation. Equation 2 provides the formula for the employment estimate ():

(2)                     

The CES program multiplies the prior month’s employment level () by the weighted link relative () and then adds the current month’s forecast value ( ).⁵

Natural disasters strain the usual assumptions about business births and deaths. Disaster conditions may halt the formation of new businesses, while forcing many existing businesses to cease operations. But despite disruptions—and sometimes even severe damage to physical locations—many businesses continue to pay employees through the hardship, mitigating the number of business deaths by CES definitions. Still, the net birth–death residual calculated from the QCEW is often noticeably lower in domains affected by natural disasters, and the forecast error tends to be positive in these cases.

Are small-area models adequate?

The CES program produces estimates with a version of the Fay-Herriot model for estimation supersectors and for many metropolitan areas with small sample sizes.⁶ The model estimate is a weighted combination of the direct weighted link-relative estimate and a synthetic component consisting of a historical trend value corrected by a coefficient derived from the regression model in equation 3:

(3)                      

Y1_i,j represents the statewide weighted link-relative estimates for a given industrial supersector (i) and state (j), Y2 represents the corresponding historical trend values,  is an estimated coefficient, and  is a model residual. The regression modeling is performed at a statewide supersector level, and the correction factor ( from the state-level model is linked to detailed metropolitan areas. The Fay-Herriot estimate for metropolitan area k is displayed in equation 4:

(4)                      .

The weight (W_i,k) assigned to the direct estimate (Y1_i,k) is a function of the variance of the direct and trend (Y2_i,k) components as well as the variance from the state-level model. When the relationship between the direct estimates and the historic trend is looser, the direct estimates receive a higher weight. The correction factor adjusts the historic trend values to the direct estimates roughly on average across the country, but it does not account for any clusters of heterogeneity among states or areas.

This model relies on assumptions about the similarity of industry behavior between areas in order to “borrow strength” across those areas; the model also relies on assumptions about the relationship between statewide and metropolitan area trends.⁷ Hurricanes bring these assumptions into stark relief, as some affected states may be dissimilar to the nation as a whole, and even within those states some areas face devastation while others remain unscathed. Similarly, the COVID-19 pandemic affected jobs in states and cities to widely different degrees, despite broad-based shutdowns and job losses.

Five major hurricanes

Although the CES survey began in 1915, and the program has faced many natural disasters since then, the survey has evolved substantially over the years. The CES program fully implemented a probability-based design for the 50 states and Washington, DC, in 2003, BLS assumed responsibility for producing the state and local estimates in 2011, and estimation procedures and models have continued to evolve since then.⁸ As a result, the 2017–18 hurricanes provided the best context in which to address estimation issues related to the COVID-19 pandemic.⁹

Several major hurricanes made landfall on the Gulf of Mexico and Atlantic coasts, as well as over U.S. territories in the Caribbean, during the 2017 and 2018 hurricane seasons.¹⁰ The timing of these storms matters because anyone who worked or received pay for any portion of the pay period including the 12th day of the month was counted as employed. Hurricane Harvey made landfall at Port Aransas, Texas, as a category-4 storm, on Friday, August 25, 2017, and produced heavy rainfall and severe flooding in coastal Texas and other parts of the Gulf Coast, most notably in Louisiana. The system that became Hurricane Irma also formed in August 2017 and made landfall over several Caribbean islands as a category-5 storm in early September; it struck the Florida Keys as a category-4 storm on Sunday, September 10, 2017, and reached the mainland later that day, continuing north along the Florida peninsula. Hurricane Maria formed in September 2017 and caused severe devastation as it made its way through the Caribbean that month. The center of the then-category-5 storm passed just south of St. Croix in the U.S. Virgin Islands on September 19, 2017, before moving diagonally across Puerto Rico the following day.¹¹ The next year, Hurricane Florence reached category-4 strength and weakened to a category-1 storm, before making landfall on Friday, September 14, 2018, near Wrightsville Beach, North Carolina; it then traveled into South Carolina, causing severe flooding and storm surges in both states. The following month, Hurricane Michael struck the Florida panhandle on Wednesday, October 10, 2018, as a category-5 hurricane.

Hurricanes Harvey and Irma both caused tragic loss of life and rank among the most severe U.S. natural disasters, in terms of property damage. Harvey caused 68 direct deaths in Texas and an estimated $125 billion in property damages in the region. Irma caused 10 direct deaths across three states—7 in Florida, 2 in Georgia, and 1 in South Carolina—and 3 direct deaths in the U.S. Virgin Islands, as well as $50 billion in property damages. The CES survey measured far fewer job losses associated with Harvey than with Irma, however. Texas employment grew in September 2017—adding 14,600 jobs, not far off the prior 12-month average growth of 16,900 jobs—with net job losses in coastal metropolitan areas such as Houston-The Woodlands-Sugar Land, TX (–8,800). The state added 39,300 jobs the following month as businesses bounced back. Florida, by contrast, lost 172,700 jobs in September 2017, before recovering 194,700 jobs the following month. The difference resulted from the timing of the storms. Irma struck Florida at the beginning of the week that included September 12, so workers on a weekly payroll who lost their shifts for 7 days showed up in the estimates as a job loss. By contrast, Harvey hit Texas more than 2 weeks earlier, so only prolonged layoffs resulted in a drop in measured employment in that state.¹² Hurricane Maria, despite hitting Puerto Rico and the Virgin Islands well before the October 2017 reference period, followed closely behind Irma’s strike on the territories and caused lasting devastation that was evident in employment data for a prolonged period.

The birth–death model

In normal months, a small number of businesses report zero employment in the CES survey. The rate of reported establishment “deaths” generally follows a stable, seasonal pattern, and some of the deaths are temporary. Following Hurricanes Irma and Maria, the number of establishments reporting zero employment increased sharply. Table 2 shows the proportion of respondents that reported zero employment and compares the forecasted and actual birth–death residuals (later derived from the QCEW) in disaster months. In order to provide frames of reference, table 2 also shows the average proportion of business deaths in the sample over the 2-year period before the disaster and the average absolute monthly forecast error, in terms of employment. The proportion of reported establishment deaths was double the usual rate in Florida following Hurricane Irma and 7 times the usual rate in Puerto Rico following Hurricane Maria. This indicated that the usual birth–death relationship did not hold, and actual birth–death values from the QCEW—calculated several months after the initial release of the CES estimates—were substantially more negative than the forecast values. The forecast error of employment in Florida following Irma was nearly 4 times the average absolute monthly forecast error, while the forecast error for Puerto Rico following Maria was more than 8 times larger than average. Following other major hurricanes, the proportion of reported establishment deaths was relatively normal, and the birth–death forecasts were closer to their mark and well within normal ranges. (See table 2.)

Although reported establishment deaths generally are not used in the estimation process, the seven-fold increase in their rate in Puerto Rico following Hurricane Maria necessitated a different approach. BLS determined the risk of underestimating the number of business deaths to be substantial enough that reported zeros were treated the same as other matched reports and were used in the sample link relative. This change negatively affected the October 2017 employment estimates by 11,800 jobs, nearly offsetting the birth–death forecast error of 13,800. Because many of the closures were temporary, the returns from zero employment were also used in the estimates.

Following Hurricane Irma, the CES program did not initially use reported business deaths in the weighted link-relative estimator for Florida, but after benchmarking the September 2017 employment data, the program recognized that the initial sample-based estimates undercounted the extent of job loss.¹³ The QCEW data contained a large increase in the number of establishments reporting zero employment, and had the sample-based estimates used the reported zeros, the estimated employment level would have been 35,000 lower—more closely tracking the population data—which would have accounted for the birth–death forecast error. (The actual value was 28,000 lower than the forecast value.) The CES program included the returns for businesses that had reported declines to zero following Irma in the sample-based estimates that were produced from the new September 2017 benchmark level, increasing Florida employment estimates in October 2017 by 32,000. Including these returns from zero and making other adjustments to account for reporting differences between the CES survey and the QCEW resulted in October–December 2017 reestimates that would more closely track the eventual benchmark data released in March 2019.

Nonresponse analysis

Following several major hurricanes, concerns arose over the possibility of businesses in severely affected areas responding at lower rates than those in the rest of the state, which would result in a violation of the missing-at-random assumption. The CES program has investigated nonresponse weight adjustments following hurricanes and other disasters such as floods and wildfires dating back to Katrina, in 2005, in order to appropriately represent disaster areas in statewide estimates. Although these adjustments can be calculated for different levels of devastation—for instance using flood maps—the set of counties designated by the Federal Emergency Management Agency (FEMA) as eligible for individual and public assistance has typically defined disaster areas for purposes of nonresponse analysis. The program now regularly monitors response rates and the impact of differential nonresponse in states affected by natural disasters. However, response rates were not significantly lower in disaster areas following the 2017–18 hurricanes, in part because of dedicated data collection efforts. Applying nonresponse adjustments following the 2017–18 hurricanes would have changed estimates little, with no obvious benefit, and therefore the estimates were not adjusted for differential nonresponse.

Small-area estimation

The CES program widely uses variants of the Fay-Herriot model in producing employment estimates, particularly at the metropolitan-area level.¹⁴ In areas affected by recent major hurricanes, the direct employment estimates were lower than the synthetic components 65 percent of the time. This indicates a breakdown in the assumptions of similarity across states and among areas within each state. Chart 1 shows a comparison of the link relatives for the direct and synthetic parts of Fay-Herriot models in metropolitan areas containing counties designated by FEMA as eligible for individual and public assistance. (Three outlier values are not displayed.) The reference line has a slope of 1, and values above that line indicate a direct estimate lower than the synthetic one.

Although the direct estimates showed the job-losing effects of these hurricanes more, on average, than did the model as a whole, the sample in these domains is small and often quite volatile. Had the CES program used direct estimates instead of Fay-Herriot models, the root-mean-square benchmark revision to the link relatives would have been 37 percent higher.

Experiences following the 2017 and 2018 hurricanes led to the development of other modeling approaches that would generate better results and not violate important assumptions. Instead of modeling all statewide supersectors together and applying the results to metropolitan areas, the domains directly affected by hurricanes could be pooled and modeled together. Had it been used, this “hurricane variant” of the Fay-Herriot model would have produced benchmark revisions that were, on average, 12 percent lower, and it would have improved 64 percent of estimates. The variant model would have improved accuracy the most for the hurricanes with the largest effect on estimates, reducing benchmark revisions for Puerto Rico following Maria by 39 percent and for Florida following Irma by 11 percent. (The hurricane variant improved estimates 71 percent of the time for Puerto Rico and 74 percent of the time for Florida.) The hurricane variant model did not perform well for Texas following Harvey: although the correction factor better captured the drop in employment, worse model fit resulted in a much larger weight on the direct components, which would have resulted in higher benchmark revisions. (See table 3.)

Applying lessons learned to producing estimates during the COVID-19 pandemic

A historically unprecedented number of jobs were lost and regained in the months after the onset of the COVID-19 pandemic.¹⁵ Health officials identified the first case of COVID-19 in the United States in Washington State on January 21, 2020.¹⁶ However, it would be several weeks before community spread of the virus led to severe business disruptions and shutdowns. Many novel data sources show declining activity throughout the month of March 2020,¹⁷ even before the issuance of formal shelter-in-place and stay-at-home orders.¹⁸ The week of March 8–14 marked a turning point in business activity. Restaurant reservation data from OpenTable illustrate the evolving decline in business activity that week: as of Sunday, March 8, reservations were down only 2 percent over the year, but by the following Saturday (March 14), they showed a 42-percent decline.¹⁹ Ohio was the first state to close all public schools on March 12, and by the end of the following week (March 15–21), nearly all public schools had closed,²⁰ air passenger travel had fallen by 75 percent,²¹ and restaurant reservations had dropped by nearly 100 percent. The National Bureau of Economic Research (NBER) Business Cycle Dating Committee determined that the longest economic expansion in U.S history ended in February 2020 and a recession began in March.²² Nationwide job losses in March (–1.7 million) and April (–20.7 million) marked the steepest employment decline in history, and were followed by the two largest monthly job gains ever in May (+2.8 million) and June (+4.8 million). The steepness and suddenness of these job losses, followed by a rapid (if partial) recovery, were more reminiscent to the losses seen after major hurricanes than those seen during a typical recession.

The birth–death model

The nationwide increases in the number of establishments reporting zero employment in March 2020 (2.6 times normal) and April 2020 (15 times normal) could only be compared with the spikes following Irma in Florida (2.1 times normal) and Maria in Puerto Rico (6.9 times normal). Evidence from Irma and Maria indicates that using reported zeros could reduce error. Beginning with the March final and April preliminary estimates, the CES state and area program incorporated “excess” reported zeros in the link relative in calculating the employment estimates. In order to avoid statistical bias, the program applied a weight-reduction adjustment value ranging between 0 and 1 to establishments reporting zero employment. In cases in which the proportion of reported zeros at the state estimation supersector level was at or below normal, the factor was set to 0, and in those cases in which the proportion was many times the usual rate, the factor approached 1. The average weight adjustment factor for a reported zero was 0.603 in March and 0.954 in April. As with Hurricanes Irma and Maria, employment counts also needed to appropriately reflect returns from zero. Weight adjustment factors were also developed for returns, and they were first applied in the May preliminary estimates, as many business reopened. The use of excess reported drops-to and returns-from zero lowered the final sum of state employment estimates by 331,000 in March and 3.0 million in April, while increasing them by 1.0 million in May and 934,000 in June, as many businesses reopened.

In addition, the CES program augmented the time-series forecast component of the net birth–death model with information from the nonzero matched sample as an exogenous regressor in a regression model with autoregressive integrated moving average errors (regARIMA), beginning with the April values.²³ Earlier research indicated that this method could improve forecast accuracy during business cycle turning points, although it was not feasible to implement following hurricanes.²⁴ The CES program updated the forecast models at the national level and controlled the sum of the state subsector forecasts to the new national forecasts, incorporating sample information on the states and industries seeing spikes in reported zero employment to allocate the distribution of differences. This lowered the sum of state birth–death factors by 799,000 in April compared with what they would have been without the regressor. The models in turn processed more positive sample information in May and June 2020 into higher forecast values.

Nonresponse analysis

The COVID-19 recession led to nonresponse concerns about the state and area employment estimates that were different from but related to those explored after hurricanes. Subdividing states into affected and unaffected regions, as is done for a hurricane, was not a promising option because of the geographic pervasiveness of business disruption. The CES program explored making differential-response-rate adjustments based on business size (the hypothesis being that larger firms may be less disrupted and better able to continue paying employees), but the results were unclear and any adjustments may have exacerbated the effects of confounding factors. The program also explored conducting nonresponse analysis based on linked unemployment insurance filings, but doing so did not uncover notable differences between respondents and nonrespondents.

In producing the April 2020 state-level estimates, a major nonresponse problem appeared as certain detailed industries within the estimation supersectors were broadly underrepresented across states in the responding CES sample in a way that correlated with employment trends. In retail trade, clothing and clothing accessories stores closed almost everywhere in April 2020, and these establishments responded to the survey at relatively low rates, while general merchandise stores and building material and garden equipment stores had comparatively resilient employment and responded at higher rates. Within leisure and hospitality, full-service restaurants lost relatively more jobs and responded to the CES survey at lower rates than limited-service restaurants, while in the “other services” supersector, the laundry and personal care services industry showed steep losses and was underrepresented in the responding sample. This nonresponse issue did not arise in the national estimates because they are produced at a detailed industry level using samples from all states, in contrast to the approach used in the state and local estimates, where estimation supersectors pool samples from heterogeneous industries into a discrete geographic domain.²⁵

The CES program took two approaches to nonresponse adjustments for these industries. If it was possible to do so, especially in very large states, the program calculated sample-based estimates at a more detailed level and summed them to replace the estimation supersectors.²⁶ In states for which this approach was infeasible, the CES program calculated nonresponse adjustments similar to those calculated (but not applied) following hurricanes and those obtained by using preidentified nonresponse factors in the link-relative estimator (d_i). Many of the response-rate differences were longstanding, but they had not substantially affected the estimates. In total, nonresponse adjustments served to lower the sum of April 2020 state employment estimates by 461,000 in leisure and hospitality, 413,000 in retail trade, and 73,000 in other services. As jobs returned, the differential nonresponse problem presented in the opposite direction, and correcting for it increased employment estimates.

Small-area estimation

The CES program explored an approach similar to the hurricane variant of the Fay-Herriot model in March 2020. The program considered a scenario in which certain states and metropolitan areas in the Northeast and Pacific Northwest that had early cases of COVID-19 might have substantially larger employment drops in March, in which case it could have been beneficial to pool those areas together and model them separately. This was not the case, however, as job losses that month presented along a continuum, with no clearly identifiable division, and were only loosely related to early case counts. The ordinary Fay-Herriot models were also problematic. The wide range of job losses among states resulted in very high model variance and subsequently near-total reliance on the direct estimates. Although the direct estimates are approximately unbiased, they tend to have high variability when sample sizes are small, and the poor performance of direct estimates following hurricanes in modeled cells indicated the potential for low accuracy.

Instead, BLS generated and in many cases used a small-area model that generalizes the Fay-Herriot model and relaxes many of its assumptions by jointly modeling variance and point estimates and clustering the data.²⁷ This model outperformed the existing Fay-Herriot model in simulations, in terms of benchmark revisions, during both the steep downturn of the 2007–09 recession and the subsequent recovery and expansion. When applied during the COVID-19 recession, metropolitan area estimates were more consistent with statewide values and showed a similar story: compared with the Fay-Herriot estimates, the new model reduced the root-mean-square difference between metropolitan and statewide estimates in the same state or industry by 29 percent in the April 2020 preliminary estimates. The combination of lessened reliance on volatile direct estimates (because of better model fit) and state-specific effects being captured in the synthetic component of the clustered model resulted in the tighter relationship among metropolitan area estimates and corresponding state-level estimates.

Conclusion

Timely economic data is needed to capture rapid shifts in business conditions shortly after they occur, and payroll employment estimates from the CES program are among the timeliest economic indicators available each month for states and metropolitan areas. Natural disasters often cause sudden, steep job losses, but the measurement of employment change following disasters presents challenges. The CES survey must accurately capture the relationship between business openings and closings, account for differences between respondents and nonrespondents, and use robust models to accurately estimate employment in small domains. Five major hurricanes that occurred in 2017 and 2018 brought these concerns to light and resulted in the development of potential solutions. Using reported zeros proved beneficial to the estimation process after the survey showed sharp increases in their number. Modeled estimates could be improved by refining the pool of observations that are processed together.

When the COVID-19 pandemic caused the steepest job losses in U.S. history, the CES program applied the lessons learned from producing estimates following major hurricanes to better capture business deaths, properly represent industries experiencing the most severe declines, and incorporate models that capture important differences between, and commonalities among, states. Using reported zeros proved beneficial following Hurricanes Irma and Maria, and the CES program generalized their use with the wave of business closures that began in March 2020. Although nonresponse was not a major problem after the 2017–18 hurricanes, monitoring processes set up in the wake of those hurricanes helped BLS identify industry-based differential nonresponse adjustments when the steepest job losses in history occurred. Techniques pooling similar observations in Fay-Herriot models showed promise, and the CES program applied a model that more formally clustered the data to produce small-area estimates during the pandemic.

Although job losses during the COVID-19 recession have been far steeper and more sudden than those occurring in typical downturns, the associated measurement challenges highlight areas of potential improvement that could help the CES survey better capture future business-cycle turning points and sudden changes in state and area employment that result from disasters.