Occupational Requirements Survey: results from a job observation pilot test

In the summer of 2012, the Social Security Administration (SSA) and the Bureau of Labor Statistics (BLS) signed an interagency agreement to begin testing the collection of occupational requirements data. As a result, BLS established the Occupational Requirements Survey (ORS) as a test survey in late 2012. The goal of ORS is to collect and publish occupational information that meets the needs of SSA at the level of an eight-digit code based on the Standard Occupational Classification (SOC) system that is used by the Occupational Information Network (O*NET).¹

The ORS data are collected under the umbrella of the National Compensation Survey, which field economists use to collect data. Field economists generally collect data elements either through a personal visit to the establishment or remotely by telephone, email, mail, or a combination of modes.

For ORS, field economists collect occupationally specific data elements to meet the needs of the SSA in the following categories:

•  Physical demands
• Specific vocational preparation
•  Mental and cognitive demands
•  Environmental conditions in which the work is performed

In fiscal year 2015, BLS completed data collection for the ORS preproduction test. The preproduction test might better be described as a “dress rehearsal” because the collection procedures, data-capture systems, and review were structured to be as close as possible to the structure of those that would be used in production.²

Background on job observation pilot test

The ORS job observation pilot test was intended to assess whether the data collected through ORS interview collection methods are systematically different from the data collected through direct observation. This test was conducted in response to both public comments in a Federal Register notice and an external subject matter expert’s recommendations for testing and validating ORS design.³

The observation test was conducted in the summer of 2015, running from June to September. The observation test involved recontacting a subset of establishments that were interviewed as part of the preproduction test. Pairs of field economists were sent to observe select jobs within each establishment and record data on the physical and environmental data elements during a 1-hour observation period.⁴ The 1-hour observation period sought to achieve a balance between gathering data that represent the job and limiting the respondent burden involved in conducting such a test.

Because the goal of ORS is to produce estimates at the eight-digit O*NET SOC level, the observation test was structured to allow us to compare preproduction data with observed data at the eight-digit SOC level as well. Thus, a subset of occupations was chosen for inclusion in the test. The subset was chosen on the basis of two criteria:

1. The occupation must appear in the preproduction microdata at least 40 times.
2. The jobs have substantial physical requirements and are relatively simple to observe (e.g., long-haul truck drivers were excluded).

These criteria resulted in the following occupations sampled for the observation test:

• Nursing assistants
• Cooks, institution and cafeteria
• Cooks, restaurant
• Waiters and waitresses
• Dishwashers
• Janitors and cleaners
• Maids and housekeeping cleaners
• Childcare workers
• Cashiers
• Retail salespersons
• Receptionists and information clerks
• Team assemblers
• Industrial truck and tractor operators
• Laborers and freight, stock, and material movers, hand

Procedures for job observation pilot test

The sample consisted of 540 jobs (456 from private industry and 84 from state and local governments) from existing ORS preproduction establishments. Establishments were sampled across geography, industry, and size to ensure a good distribution of available establishments within each occupation.

For each of the sampled establishments and occupations, a field economist secured the appointment and explained to the respondent the reason for the followup visit. A pair of field economists then collected data by way of a personal visit. The paired field economists, if possible, simultaneously observed the same employee in a preselected job and documented the situation for a 60-minute period. However, if pairing was not possible, one economist would observe a separate employee. The field economists were instructed not to look at data recorded from the preproduction test for their establishments or to discuss their data with one another. Each field economist independently recorded and coded his or her observations during the personal visit. Field economists were as inconspicuous as possible and did not ask questions of the observed employee.

The field economists were instructed to code the duration in minutes and to code a duration of zero if the element was not observed. Some elements had additional questions, such as, Was the job performed with one hand or both? For these elements, each field economist checked the appropriate box category and noted the duration in minutes. Field economists then checked their data for accurate recording before marking the schedule and quote as complete. The team overseeing the test held two collection debrief meetings with the field economists (one midtest and one at the end of collection) to assess how the process worked.

Response rates

Field economists contacted establishments for 405 of 540 jobs in the sample and observed 244 jobs, a 60-percent response rate.⁵ As shown in table 1, the refusal rate varied by occupation.

Childcare workers and nursing assistants had very high refusal rates. These refusals largely stemmed from establishments’ concerns about privacy under state and national laws. Some successful observations of these two occupations occurred during the observation test; however, because of the small sample size, we did not include any childcare workers or nursing assistants in our test analysis.

Measures of agreement between preproduction and observed duration

We evaluated the agreement between the observed values of the data elements and those collected (during the interviews) in the preproduction test. We will refer to these as “observed” and “interview” values hereon.⁶ Our analysis focused on the physical elements defined in table 2.

The durations of most physical elements for preproduction were classified into five categories:

1. Not present
2. Seldom—up to 2 percent of the day
3. Occasionally—2 percent up to one-third of the day
4. Frequently—one-third up to two-thirds of the day
5. Constantly—two-thirds or more of the day

Measuring agreement between observed and interview data was complicated by two factors:

1. The duration of the observation test was short, which may lead to discrepancies between the presence or absence of certain physical requirements. In particular, we expected high degrees of agreement in the presence or absence of physical requirements with durations that fall into the “frequently” or “constantly” categories and low degrees of agreement for elements that occur “occasionally” or “almost never.”
2. In preproduction collection, roughly 20 percent of the physical requirements that were classified as “present” in the job had no duration provided by the respondent. The unknown duration is especially high in particular elements—in the sample of jobs that were observed, the interview data had missing duration in nearly 30 percent of the cases for communicating verbally and 25 percent of the cases for fine manipulation.

To address the challenges posed by the short duration of the job observation test, we recategorized the durations into four categories, aggregating the classifications “not present” and “seldom” into one category:

1. Not present or seldom—less than 2 percent of the day
2. Occasionally—2 percent up to one-third of the day
3. Frequently—one-third up to two-thirds of the day
4. Constantly—two-thirds or more of the day

First, we calculated “raw” levels of agreement between the observed data and interview data. These levels are presented in column 2 of table 3. The levels of agreement are relatively high, ranging from 71.2 percent for reaching at or below shoulder level to 97.4 percent for pushing and pulling with feet.

The levels of agreement reported in table 3 suggest that the durations of physical requirements were similar across modes of collection, but statistical tests of agreement were required to ensure that the levels of agreement did not occur by chance. Therefore, we used a weighted version of Cohen’s kappa to assess agreement across modes of collection.⁷ The kappa statistic measures the agreement against a benchmark of the expected agreement. Bear in mind, however, that if only a few possible categories exist, the observed and interview data could fall into the same duration category simply by chance. The weighting in our kappa measure penalized less for disagreements that were close (e.g., the observed duration was “frequently” and the duration from the interview was “constant”) and more severely for disagreements that were farther apart (e.g., the observed duration was “not present or seldom” and the duration from the interview was “constantly”).

Kappa generally ranges from ‒1 to +1. Negative values of kappa indicate that the level of agreement is less than the expected agreement. Kappa statistics close to 1 imply a higher level of agreement. Although some controversy exists in the literature regarding thresholds of kappa, J. Richard Landis and Gary Koch have proposed the following standards:⁸

• Less than or equal to 0 is poor agreement.
• 0.01–0.20 is slight agreement.
• 0.21–0.40 is fair agreement.
• 0.41–0.60 is moderate agreement.
• 0.61–0.80 is substantial agreement.
• 0.81–1.00 is almost perfect agreement.

As can be seen in column 3 of table 3, however, the expected levels of agreement are relatively high and the kappa statistics are relatively low.⁹ With the exception of crawling, crouching, and pushing and/or pulling with feet, the agreement levels are greater than the expected levels of agreement in a 5-percent one-tailed test (column 5 of table 3). The kappa statistics vary considerably; the average value of the kappa statistic is 0.20, which denotes relatively low levels of agreement relative to agreement by chance.

A well-known issue with kappa is the influence of prevalence and bias on the kappa measures. Generally, variables with underlying uniform distributions result in higher values of kappa. Measuring kappa using data that have skewed prevalence can cause the “kappa paradox,” in which high levels of rater agreement have relatively low kappa statistics.¹⁰ The distributions of the physical elements in ORS, however, tend to be highly skewed (in terms of both underlying discrete values of the duration as well as the categorical measures). As figure 1 illustrates, the distribution of observed duration for two physical elements—crouching and gross manipulation. The mode for crouching is “not present or seldom,” which was recorded in over 70 percent of the observed jobs. Frequent or constant crouching rarely occurs. On the other end of the spectrum, gross manipulation is constant for most of the observed jobs, which is not surprising when one considers the occupations sampled (cashier, maid, etc.).

View Chart Data

To reduce the impact of skewness on our test statistics, we used a measure of prevalence-adjusted and bias-adjusted kappa (PABAK). The PABAK measure is presented in column 6 of table 3. The average is 0.68, which is considered “substantial”; however, the PABAK measure varies considerably by data element. In particular, stooping, reaching at or below the shoulder, communicating verbally, fine manipulation, gross manipulation, and pushing and/or pulling with hands and arms have low measures (less than 0.50), even after prevalence and bias are adjusted.

Of particular concern, given the potential uses of ORS data in the disability determination process, was whether the preproduction interview data appear to understate the duration of the physical elements, especially for those elements with relatively low PABAK measures. We evaluated potential under- or overstatement of duration using a sign test. The sign test is a test of the difference in medians. We were particularly interested in elements in which the sign test rejects the null hypothesis that the observed median is less than or equal to the interview median—rejecting this hypothesis implies that the observed durations were the durations collected through interviews (see column 2 of table 4).

The elements with longer duration associated with observation are stooping, reaching at or below the shoulder, other keyboarding, fine manipulation, gross manipulation, pushing and/or pulling with hands and arms, and pushing and/or pulling with feet. When we measured the modes of these elements, only one differed in mode between the collected and observed values—reaching at or below the shoulder level. The value of the mode for this element was “occasionally” (2 percent up to one-third of the day) in the interview data and “constantly” in the job observation data (two-thirds or more of the day).

Missing duration was identified as an issue with ORS preproduction data for some of the physical elements. In the case of reaching at or below the shoulder level, 53 of the job observation duration measures could not be compared with interview duration data because of missing duration. Notably, among the 53 missing quotes in preproduction, 64 percent of the quotes of the job observation test recorded durations of frequently or constantly. This result was a common pattern among those elements in which the sign test rejected the null of observation duration equal to or below preproduction—the missing data in preproduction aligned with observed durations above the median and mode.

Conclusions

The job observation pilot test validated the ORS physical elements by comparing the data collected through preproduction interviews with those data collected through a different source—observation. Pairs of field economists were assigned to observe the same job for 60 minutes and record the duration of each physical element of the job.

For elements that workers performed infrequently, the 60-minute observation period may have led to more disagreement between observed data and interview data collected during preproduction. The PABAK measures of duration were relatively strong, suggesting that the data collected during interviews and observations had high levels of agreement across most elements.