This is one more installment in a continuing series, brought to you by the universe, entitled "promising pilot projects often don't scale". They don't scale for corporations, and they don't scale for government agencies. They don't scale even when you put super smart people with expert credentials in charge of them. They don't scale even when you make sure to provide ample budget resources. Rolling something out across an existing system is substantially different from even a well run test, and often, it simply doesn't translate.
Sometims the "success" of the earlier project was simply a result of random chance, or what researchers call the Hawthorne Effect. The effect is named after a factory outside of Chicago which ran tests to see whether workers were more productive at higher or lower levels of light. When researchers raised the lights, productivity went up. When researchers lowered the lights, productivity also went up. Obviously, it wasn't the light that boosted productivity, but something else--the change from the ordinary, or the mere act of being studied.
Sometimes the success was due to what you might call a "hidden parameter", something that researchers don't realize is affecting their test. Remember the New Coke debacle? That was not a hasty, ill-thought out decision by managers who didn't care about their brand. They did the largest market research study in history, and repeated it several times, before they made the switch. People invariably told researchers they loved the stuff. And they did, in the taste test. But they didn't love the stuff when it cost them the option of drinking old Coke. More importantly, they were being offered a three-ounce cup of the stuff in a shopping mall lobby or supermarket parking lot, often after they'd spent an hour or so shopping. New Coke was sweeter, so (like Pepsi before it) it won the taste test. But that didn't mean that people wanted to drink a whole can of the stuff with a meal.
by Meagan McArdle, The Atlantic | Read more:
Photo: Paxton Holley via Flickr
