We are using three primary measures to assess the performance of our markets: accuracy, stability, and timely response to genuine demand shifts. Having run pilot markets for approximately 18 months, we are starting to get a sense for how the markets are performing. Although the market forecasts and official company forecasts are not independent, it is nonetheless interesting to compare the signals and then assess how effectively they are working together. In terms of accuracy, the markets are producing forecasts at least the equal of the official figures and as much as 20% better (20% less error), an impressive result given that the official forecasts have set a rather high standard during this time period with errors of only a few percent. In the longest sample to date, six of eight market forecasts fell within 2.7% of actual sales. The accuracy of the official and market forecasts has been remarkably good, well within the stated goal of +/- 5% error for all but a few individual monthly forecasts. Until more results are generated over time we will not be able to determine the extent to which this strong performance stems from the introduction of the market forecasting process. It is also possible that sales were unusually easy to forecast. Regardless, specific results from the pilots have shown the value of the market forecasts and are leading us to believe the markets are having a positive impact.

On one occasion we saw the first market for an upcoming quarter's sales vote "no confidence" on the prior official forecast. Ranges of potential sales in the IAM are structured so that the prior official forecast is roughly centered in the set of ranges. A "no confidence" vote occurs when all investments from participants come in either above or below that official forecast, meaning that the group believes there is a 100% chance of falling on one side of the official forecast. The only time this has occurred the market forecast was correct. The official forecast published prior to the market forecast was off by over 10%, and the market led it in the right direction.

Much like public stock markets, we have seen our IAMs react quickly and decisively to strong news and then take time to assess and properly discount it. One IAM dropped by 4.7% and then bounced back to almost exactly where it was before the drop. This was not accidental. A rash of cancelled orders and bad news that appeared to signal softening demand turned out to be an aberration, and the market needed time and additional information to make that call. These sorts of sudden shifts are unusual. In fact, the IAM forecasts are quite stable, with as much as 20% less fluctuation from month to month than the official forecasts during the same period. The business planning team responsible for the official forecast observed that the market signals were more stable and implemented a new process step to try to filter noise from each new official forecast.

Surprisingly, the market forecasts are not necessarily improving as the forecasting horizon shrinks. Although we will need a longer history of data to draw a firm conclusion, we have some evidence that the forecast is as likely to get worse in the final month before the actual result is known as it is to get better. The reason, as we understand it today, is that as the amount of signal goes up rapidly toward the end of the period, the amount of noise goes up rapidly as well. As the amount of information explodes and the time to assess it shrinks, it would not be a surprise to see humans unable to tell the forest from the trees. Fortunately, forecasts out in the 3-8 month horizon, which provide the factories ample opportunity to plan product starts, are performing quite well.

Another key set of results is feedback from owners of the official forecasts, as well as market participants. Discussion with the owners has centered around learning to produce better official forecasts from the market results. The value or credibility of the results has never been questioned; in fact, the one month we were late publishing the market results brought reminder e-mails from the owners. Not long into the pilots the owners began discussing new markets for other key forecasts. Clearly, they are seeing the value of this new data source. Participants have been quite positive as well. Quotes such as "I enjoy participating in the trials" are common. Another trader cited the IAM process as a welcome break from the often mundane job of forecasting: "I think it's great we're doing this simply because it makes work more fun and incentivizes us to do our homework and make the right call, which should lead to better results." We are also amused that although we never publish the list of participants and winners, everyone knows who participated and who won.

Based on the results and word-of-mouth advertising, interest in expanding the research into new parts of the business is growing. We expect the number of forecasting markets to quadruple in the next three months. More implementations producing more data will accelerate our pace of learning.