Like many good arguments, this one started over a stiff drink. An Earl Grey MarTEAni, to be precise.
In January 2010, Nathalie Louissaint, a New York City health inspector, visited Pegu Club, an upscale cocktail bar. She watched as the bartender mixed the signature tea-infused drink. Borrowing a technique from the nineteenth century, the bartender added raw egg whites, which give the drink a silky body and an alluring layer of foam. Louissaint decided that the raw egg warning on the menu was insufficient and cited the bar for a health code violation.
The citation outraged many. Paul Clarke, a Seattle-based food writer, was perplexed by the department’s rigid position on raw eggs, writing on the website Serious Eats, “Does this mean the health department will begin targeting restaurants that serve raw eggs in a Caesar salad?” Others decried the health department’s seeming mandate to use pasteurized eggs, but those, said Pegu Club owner Audrey Sanders, “impart this really funky wet-diaper nose.” One bartender, who insisted on anonymity for fear of reprisal, told the New York Times, “If they make it illegal to serve egg-white drinks, that would be Hurricane Katrina for us.” In response to the uproar, the health department overruled the inspector.
This confusion is no outlier. Nationwide, implementation of health codes varies dramatically across inspectors and health departments. In Seattle, two inspectors observed Caesar salad dressing prepared with raw (unpasteurized) eggs in the same restaurant, but disagreed about whether to cite a violation. Contrary to New York City health department guidelines, New York State’s website doesn’t mention menu warnings, instead admonishing, “Consider using commercially pasteurized eggs in recipes that use eggs or consider removing the item from your menu.” The Centers for Disease Control and Prevention (CDC) document that 80 percent of restaurants nonetheless use unpasteurized eggs.
When it comes down to it, the marTEAni fight is not so much about eggs as it is an endemic challenge across government. From airport security checkpoints and routine traffic stops to home construction permits, citizens and government interact frequently through individual officials. At times, the decisions of these frontline government officials can seem disturbingly arbitrary.
A bar can always take a drink off its menu, but sometimes the arbitrariness can have more serious impact. In 2013 the Administrative Conference of the United States reported that administrative law judges grant Social Security Disability claims at rates ranging from 4 percent to 98 percent. In asylum adjudication, New York immigration judges vary in their grant rates from 6 percent to 91 percent when cases are assigned irrespective of merits, leading to a denunciation of the process as “refugee roulette.” A study of Illinois child-welfare case managers found substantial differences in decisions to place children in foster care based on allegations of abuse or neglect. It is no wonder that scholars assail the child-welfare system as a form of institutional “chaos, oppression, and tragic ineffectiveness.” In nuclear safety, violation-detection rates can vary from less than 10 percent to more than 60 percent depending on the inspector. The regulatory requirements are so complex that one nuclear official conceded, “Nondetection is endemic.”
Inconsistency breeds mistrust. A city analyst in New York described the city’s restaurant inspection system as “arbitrary” and concluded, “If we can’t trust the Health Department to provide real scientific data . . . then we can’t trust any agency.” Businesses feel this too. An owner of a fast-food chain with many locations across Washington state observed sharp differences in how individual restaurants were scored and how the same restaurant was scored over time, despite his chain’s uniform food-safety protocol: “We always thought we were doing a great job in putting safety first; but it turns out in many, if not most, cases, inspectors were either not being as thorough as they could have been or were only verbally coaching” rather than writing violations. One Virginia bartender, responding to the Pegu Club episode, said, “I’m not 100 percent sure what the law is.” At a staff meeting in King County, Washington, health inspectors expressed similar misgivings. They hoped for better consistency in order to build “credibility and trust.”
The pervasiveness of these challenges leads some to point fingers, some to throw up their hands, and others to bemoan government altogether. If you are on the right, blame public sector unions, civil service protections, or listless bureaucrats. If you are on the left, blame underfunding, deregulation, or the lack of federal oversight. Governments generally cannot resolve these underlying ideological battles, but they must find ways to address the consistency of frontline decision making nonetheless. The question is, how?
The Peer Review Experiment
One possible solution is peer review. If frontline government officials could review and deliberate over each other’s work, the quality and consistency of decision making might improve. While isolated examples of such peer review exist, we have regrettably little systematic evidence of peer review’s effectiveness in the public sector. The reason is understandable. Due to perceived costs, logistics, and ethical and political concerns, rigorous experiments can be difficult to design and implement when it comes to regulation.
Beginning in 2014, we designed a randomized, controlled trial to test the effectiveness of peer review with the food safety staff of King County, where Seattle is located. Half of the inspection staff was randomly assigned to engage in peer review. For sixteen weeks, these inspectors spent one day per week with a randomly selected fellow inspector, taking turns conducting inspections and independently scoring health code violations. We then used information from these peer inspections to identify and train for violations that cause the most confusion.
The results were remarkable.We discovered that, when observing identical conditions in restaurants, health inspectors disagreed nearly 60 percent of the time.
by Daniel E. Ho and Becky Elias, Boston Review | Read more:
In January 2010, Nathalie Louissaint, a New York City health inspector, visited Pegu Club, an upscale cocktail bar. She watched as the bartender mixed the signature tea-infused drink. Borrowing a technique from the nineteenth century, the bartender added raw egg whites, which give the drink a silky body and an alluring layer of foam. Louissaint decided that the raw egg warning on the menu was insufficient and cited the bar for a health code violation.
The citation outraged many. Paul Clarke, a Seattle-based food writer, was perplexed by the department’s rigid position on raw eggs, writing on the website Serious Eats, “Does this mean the health department will begin targeting restaurants that serve raw eggs in a Caesar salad?” Others decried the health department’s seeming mandate to use pasteurized eggs, but those, said Pegu Club owner Audrey Sanders, “impart this really funky wet-diaper nose.” One bartender, who insisted on anonymity for fear of reprisal, told the New York Times, “If they make it illegal to serve egg-white drinks, that would be Hurricane Katrina for us.” In response to the uproar, the health department overruled the inspector.This confusion is no outlier. Nationwide, implementation of health codes varies dramatically across inspectors and health departments. In Seattle, two inspectors observed Caesar salad dressing prepared with raw (unpasteurized) eggs in the same restaurant, but disagreed about whether to cite a violation. Contrary to New York City health department guidelines, New York State’s website doesn’t mention menu warnings, instead admonishing, “Consider using commercially pasteurized eggs in recipes that use eggs or consider removing the item from your menu.” The Centers for Disease Control and Prevention (CDC) document that 80 percent of restaurants nonetheless use unpasteurized eggs.
When it comes down to it, the marTEAni fight is not so much about eggs as it is an endemic challenge across government. From airport security checkpoints and routine traffic stops to home construction permits, citizens and government interact frequently through individual officials. At times, the decisions of these frontline government officials can seem disturbingly arbitrary.
A bar can always take a drink off its menu, but sometimes the arbitrariness can have more serious impact. In 2013 the Administrative Conference of the United States reported that administrative law judges grant Social Security Disability claims at rates ranging from 4 percent to 98 percent. In asylum adjudication, New York immigration judges vary in their grant rates from 6 percent to 91 percent when cases are assigned irrespective of merits, leading to a denunciation of the process as “refugee roulette.” A study of Illinois child-welfare case managers found substantial differences in decisions to place children in foster care based on allegations of abuse or neglect. It is no wonder that scholars assail the child-welfare system as a form of institutional “chaos, oppression, and tragic ineffectiveness.” In nuclear safety, violation-detection rates can vary from less than 10 percent to more than 60 percent depending on the inspector. The regulatory requirements are so complex that one nuclear official conceded, “Nondetection is endemic.”
Inconsistency breeds mistrust. A city analyst in New York described the city’s restaurant inspection system as “arbitrary” and concluded, “If we can’t trust the Health Department to provide real scientific data . . . then we can’t trust any agency.” Businesses feel this too. An owner of a fast-food chain with many locations across Washington state observed sharp differences in how individual restaurants were scored and how the same restaurant was scored over time, despite his chain’s uniform food-safety protocol: “We always thought we were doing a great job in putting safety first; but it turns out in many, if not most, cases, inspectors were either not being as thorough as they could have been or were only verbally coaching” rather than writing violations. One Virginia bartender, responding to the Pegu Club episode, said, “I’m not 100 percent sure what the law is.” At a staff meeting in King County, Washington, health inspectors expressed similar misgivings. They hoped for better consistency in order to build “credibility and trust.”
The pervasiveness of these challenges leads some to point fingers, some to throw up their hands, and others to bemoan government altogether. If you are on the right, blame public sector unions, civil service protections, or listless bureaucrats. If you are on the left, blame underfunding, deregulation, or the lack of federal oversight. Governments generally cannot resolve these underlying ideological battles, but they must find ways to address the consistency of frontline decision making nonetheless. The question is, how?
The Peer Review Experiment
One possible solution is peer review. If frontline government officials could review and deliberate over each other’s work, the quality and consistency of decision making might improve. While isolated examples of such peer review exist, we have regrettably little systematic evidence of peer review’s effectiveness in the public sector. The reason is understandable. Due to perceived costs, logistics, and ethical and political concerns, rigorous experiments can be difficult to design and implement when it comes to regulation.
Beginning in 2014, we designed a randomized, controlled trial to test the effectiveness of peer review with the food safety staff of King County, where Seattle is located. Half of the inspection staff was randomly assigned to engage in peer review. For sixteen weeks, these inspectors spent one day per week with a randomly selected fellow inspector, taking turns conducting inspections and independently scoring health code violations. We then used information from these peer inspections to identify and train for violations that cause the most confusion.
The results were remarkable.We discovered that, when observing identical conditions in restaurants, health inspectors disagreed nearly 60 percent of the time.
by Daniel E. Ho and Becky Elias, Boston Review | Read more:
Image: Even More Bureaucrats (1993) by Synnøve Anker Aurdal / photo by Bosc d'Anjou
