The first steps toward implementing a new IAM are finding business problems to address and teams interested in gathering better intelligence to solve those problems. In the context of demand forecasting, we started by partnering with two teams responsible for developing forecasts for product families. We determined that quarterly unit sales–with rules to define exactly which sales are included or excluded–would be useful to forecast with an IAM. With agreement on the result to be forecasted, the design process begins.

We have found through the development of current and upcoming IAM implementations that design considerations can be organized into five categories: interface, information, incentives, integration, and inclusion. The Sidebar "Five categories of considerations for designing Information Aggregation Mechanisms" lists examples of design questions that should be evaluated within each category. Since we have found that design choices in one category often depend on choices in other categories, the five categories are developed more or less in parallel. Many companies implementing markets may start by designing the interface or simply assume that the only available IAM mirrors the stock market with regular, continuous trading periods and double auction trading. In fact, many interfaces exist, and choosing the best one should be guided by many other considerations. To demonstrate the application of the five categories of design considerations, the remainder of this section covers the design process for our original pilot market.

We began our design process by considering inclusion. As a first experiment we wanted to enable the central business planning team that creates the official forecasts to generate a collective forecast using an IAM. We would compare their collective forecasts created through the IAM to their current and historical collective forecasts created through Intel's standard processes. We invited this team, and other participants who had a global perspective on the business and function more as analysts rather than sales or marketing staff. It was a relatively homogeneous pool of experts (but not without differences of opinion) and about as unbiased a group as we could put together, and we felt it would be a good baseline for future experiments that would tend toward greater diversity and bias. The total pool invited numbered from 20 to 25.

We carefully weighed how the IAM process would integrate with the workflow and processes of our participant pool. Knowing that the official forecasts are published monthly and that the potential participants are quite busy with that process for nearly two weeks out of the month, we decided against a continuous market; instead, we elected to time a snapshot IAM at the midpoint between official forecast publication dates. This scheme would maximize participation while effectively doubling the beat rate of new forecasts. The official and IAM forecasts would leapfrog each other, each outcome feeding the other process roughly two weeks later. (Since having the official forecasts and IAM forecasts influencing each other was unavoidable, we decided at least to make their interaction systematic.)

Structuring the information in the market was simple, as we decided to mirror the structure of the regular forecast. Each market would create separate forecasts for unit sales of a product family in the current quarter, the next quarter, and the quarter after that. A packet to be sent to all potential participants before each market was developed. It included the definition of the results to be forecast, how incentives would be awarded, instructions for using the forecasting application, the current official forecast, and a small set of (already available) information such as historical sales and current orders. The market interface itself would provide some information during and after each snapshot. Once actual results were determined, prizes would be announced to individual winners, and a list of prizes awarded, showing amounts but not recipient names, would be published to the whole participant group. At no point would lists of participants be published, giving all participants the option of anonymity.

The interface design was based on the experience of our academic partners and the design choices in the above sections. We knew which information we wanted to collect and that we wanted to use a monthly snapshot to collect it, so the team opted for a synchronous Web-based application that seemed a good fit. It is essentially a survey mechanism that enables each participant to create a probability distribution of unit sales while watching others enter their distributions. Participants can learn from the aggregate forecast of the group while continuing to invest their own individual budgets into the offered investments, each corresponding to a range of potential unit sales. This method had demonstrated solid results in laboratory experiments outside Intel and can develop a complete forecast in as little as 15 to 20 minutes.

The behavior of participants in the IAM is based on the way incentives are awarded. Once the actual result is known, investments made in the range containing the result are placed in a drawing for cash prizes. Each participant's chances of winning prizes are proportional to his or her share of all investments in the winning range. We wanted to avoid extremes of all incentives going to a single winner or dividing incentives among all winning tickets, because we did not want to encourage participants to concentrate their investments too narrowly or spread them too broadly. Incentives were a hot topic in the design phase–how large should they be? In the end we settled on an amount significant enough to attract and retain interest (we hoped) but not large enough that employees might shirk other job responsibilities.

Our overall design structures each investment as a decision based on both the individual's expectations for the outcome and the aggregate group prediction. Participants weigh owning lower percentages of more likely outcomes against higher percentages of less likely outcomes. In the end, we believe the system works well because having each participant weigh the conditional probabilities of various outcomes creates a robust collective forecast. And, the final outcome of the market–the amount of investment in each potential range of unit sales–forms a probability distribution based on the intelligence of the entire group.

We analyze not only the collective forecast but also the transaction records of individual traders. Assessing trading behaviors and inferring strategies from the behaviors helps us understand how the systems work, under what conditions they might work well, and why certain types of participants or investment strategies may contribute more or less toward a good (or bad) outcome. Over time we also expect to use the observed data to determine whether the formal outcome is the best possible forecast the market had to offer. Perhaps other information based on the demonstrated knowledge and track records of participants, individually or grouped by function, geography, or experience, will lead us to be able to handicap traders by the knowledge they impart to the system over time.