Imagine a commercial fishing boat out on the sea. The crew casts a large net into the water and pulls up an enormous load of fish – all kinds, all sizes. Of course, despite the size of the bounty, there are some fish that escape the clutches of the net. And even though you’ve got a tasty catch, some of the most succulent fish may be among the ones that got away.

How do you get those fish that got away?

Now, think of the net as your company’s data warehouse. The warehouse has a wealthy catch of information related to your business. But there are also other bits and pieces of information – ad hoc data sources, or "data nuggets" – that do not reside in the warehouse, but are critical to your operations. This data and knowledge includes information contained in ASCII files, Excel spreadsheets, Access databases, legacy systems that didn’t quite make the data warehouse list, departmental SQL servers and dozens of other disparate data sources.

There are four "classes" of ad hoc data sources:

• Information that is too small or "out of place" for the data warehouse – it is a minute amount of data that gets used for a specific application and is then thrown away, typically an Excel spreadsheet.
• Information that will be part of the data warehouse, but the planning for its inclusion has not been completed, or the financial resources are not immediately available.
• Information that resides in a legacy system – and is serving a useful purpose – but will never be part of the data warehouse because the cost or effort of getting it into the warehouse is too great.
• External ad hoc data sources – the tremendous amount of data that we are exchanging with our customers and vendors. These ad hoc data sources rarely get inserted into the data warehouse.

There are business intelligence tools that are capable of accessing these ad hoc data sources and integrating them into the data warehouse. But how quickly and effectively can they perform this function, and at what cost? The answers are not very quickly, not very effectively, and often at significant cost. This problem is magnified when you consider the rate at which these data sources are appearing.

Enter BII, a method of focusing the technology of business intelligence to incorporate ad hoc data sources as seamlessly as possible. The inability of traditional BI tools to deal with this data is a major problem. This key element of the total integration process was missing; thus, the emergence of BII.

Industry experts also understand the need for BII, as evidenced by a statement from IT analyst firm, META Group, in a Business Week story dated June 19, 2000: "The currency of the digital marketplace is intelligence, but it’s hard to leverage the cumulative knowledge collected by disparate systems… A real-time, panoramic, single customer view (of data) is the Holy Grail … everyone is searching for it."

Consider this scenario about the tangible benefits of BII technology: your company’s vice-president of marketing comes to you on Friday afternoon with a spreadsheet of 5,000 names that his advertising agency has provided, perhaps from a focus group, relating to customer spending practices and trends. He has jotted some additional information of his own alongside the names. He’d like to target these customers and prospects through a highly focused marketing effort, integrating this information with sales and marketing data in the data warehouse and producing a report addressing the viability of marketing to these prospects. And he wants it that same afternoon.

You might be able to produce this report through a complicated Visual Basic program or by throwing a team of engineers at the problem, but how cost-effective would that be? Ultimately, the question is, "With your existing BI tools, can you produce a meaningful report that your VP needs within hours, using this ad hoc data?" Probably not. But using BII, you would be able to quickly create the requested report integrating the ad hoc data source with the data in your warehouse, a process that would ordinarily have taken days or weeks.

The ability of BII to incorporate data from ad hoc data sources is also helpful during mergers and acquisitions. A firm that acquires another organization may want to analyze a particular product line from the acquired company, integrating legacy data with its own data warehouse to determine marketability and potential market share. Obviously, the information would reside in a separate data source, yet it must be integrated with the acquiring company’s market information to produce a meaningful report.

Perhaps the information from disparate sources will eventually present itself so it is practical to enter it into the data warehouse. The problem is that the cost and effort required to enter every last piece of information into the warehouse begins to show diminishing returns.

Meanwhile, BII is moving toward automation and pro-activity. The ideal scenario is that the information a person wants is delivered automatically, before anyone even asks for it. What’s more, it can be delivered through the device of choice, such as a pager, e-mail, cell phone or customized web portal. The idea is that the system will pro-actively investigate business anomalies, flag them, and report them in much the same way a person is automatically notified when their stock price increases and they are paged with that information. This kind of scenario is already happening, for example, when a person receives an e-mail saying that their airline ticket price has fallen within their desired price range. Why can’t this be done with your business data as well? It can be, because BII technology automates many of the engineering processes that were previously done manually.

Many Fortune 500 companies in the consumer package, utilities, computer, and education industries have already embraced BII and are enjoying business benefits. But how long will it take before other companies come to realize a need for BII? And, when they do, will they have already fallen behind?