Saturday, August 11, 2007

Do organizations "test" the decisions they make?

When his firm decided to introduce a Christmas menu into their restaurant chain, one of my students decided to test the profitability of the change by introducing it in only half the restaurants in his territory. By comparing sales changes at these restaurants (the "experimental group") to changes at restaurants that did not introduce the menu (the "control group"), he concluded that the change did little to increase overall sales, despite the apparent popularity of the menu.

This inference was possible only because my student constructed what economists call a difference-in-difference estimate of the change. The first difference is before vs. after introduction of the menu; the second difference is between the experimental and control groups.

The difference-in-difference methodology controls for other unobserved factors that might have accounted for the change. The FTC has released a number of studies following up on merger enforcement decisions to try to figure out whether they did the right thing. For two consumated oil mergers, FTC economists Dan Hosken and Chris Taylor (article) and John Simpson and Chris Taylor (article) found that prices in cities affected by the merger did not increase relative to prices in control cities. In the time series graph below, the three lines represent gas prices of the experimental city (Louisville) relative to gas prices in three control cities (Chicago; Houston; Arlington, VA) for the Marathon Ashland gasoline merger. The vertical line represents the date of the merger. By comparing prices before and after the merger, we see the merger had no effect, or that the FTC was correct to let it through without a challenge.
FTC General Counsel (now Commissioner) Bill Kovacic coined the term "Enforcement R&D" to describe the practice of government agencies following up on their decisions to improve policy (article).

I would like to hear stories from readers about how, or if, their organizations test decisions.

4 comments:

  1. I used to work at Beazer Homes, one of the nation's largest homebuilders, with billions of dollars in sales. They would have specials every quarter, designed by the marketing department, but never go back and check to see if the increased volume justified the cost of the promotions.

    ReplyDelete
  2. Check out Banerjee, Duflo, and Mullainathan's Poverty Action Lab at MIT for the case that all public policies should be tested in randomized controlled trials before being implemented. A bit more ambitious than the kinds of examples you're looking for, but interesting nonetheless!

    ReplyDelete
  3. I have never seen my company check the sucess of any new programs. We rarely even allow enough time for a new program to take effect. We tend to react in a "knee jerk" fashion. We implement a new program and then when an executive doesn't see immediate results we implement another change. So really we operate in a state of flux all the time.

    ReplyDelete
  4. Nah, I strongly doubt it, but I don’t care really much at all. As I don’t prefer these things and I prefer working straight forward stuff and that’s why I enjoy Forex trading, it is absolutely awesome given there is support for me in shape of OctaFX, it is absolutely brilliant broker to work with due to their wide range of benefits especially the low spread starting from just 0.2 pips to high leverage up to 1.500 plus much more!

    ReplyDelete