1. how long to do a study

2. I believe people do perform differently when they are under scrutiny, much as the Hawthorne Effect states. Recently, I rolled out a "Summary Report" to my team. It is a monthly report that tracks everything I view during the month and then summarizes that data and presents it as numerical findings. In some respects, it looks like a Report Card. After announcing this new way of viewing performance data, we noticed drastic decreases in the number of errors on certain applications. Although I had been viewing the data all along, I never put it on paper for everyone to see and compare. The changes are interesting. Hopefully the positive swing continues.

3. I have done something quite similar in my job. Working in retail, it is vital to the store's success to track performance of each department to see where each one is tracking in reference to the store as a whole, to other departments, and to themselves as compared to pervious months. I started tracking the out of stock (how much product we DON'T have on our shelves) as well as our sales in each of the departments for each week. I created a spreadsheet so that the numbers could easily be looked at and compared to other stores in the market and other departments within my store. Like your situation, I saw a huge decrease in the amount of out of stocks (this is good!!) the first few weeks I was tracking this, and an increase in sales. [obviously out of stock decreasing increased our sales - huge correlation there] Unfortunately the trend leveled off considerably after the first few weeks. Not sure if it's because the "newness" of this new program wore off, or because people became less interested. Of course, I've seen this happen in other areas before, so I sort of expected it to happen, although it didn't stop my disappointment. I think perhaps it's human nature to find out information like this and to want to change it to make it better, but over time that drive and desire begins to fade and deminish.

4. More scary...considering the store you work for...why isn't that a standard part of every manager's job? It also does show how when you set up a research project, you have to make sure it is long enough to reveal what you want to find out. A 2 week trial might show wonderful drops in out of stock items...so senior management rushes to implement it at all stores. But a 6 month study shows the out of sticks drop reverts back to the previous baseline around 4 months (I hope not). Many studies show this type of too-short-of-a-study problem, but they get picked up by policy makers since they do show great improvements. But rolling out on a wider scale fails. Or they worked because of the personality of the person being studied and the research study missed that...and that is a problem with case study research.