Donald Farmer is the Principal of TreeHive Strategy and formerly the VP of Innovation at Qlik.
So you have chosen a platform for Business Intelligence: flexible, usable, manageable, scalable: all those great abilities in one package. Soon you'll integrate, visualize and collaborate across the company as never before. A consistent view of the data is at hand. You can see the end of spreadsheet hell. No more complaints about compliance. Except ...
It never quite works that way, does it? Your carefully crafted architecture strains to hold together from the first day of deployment.
At 10pm, the night before the quarterly business review, with numbers to crunch and new charts to create, your sales team still reaches for Excel rather than your shiny BI tool. Convenience wins over compliance, and years of practice makes Excel the easy choice. Nor should you think this scenario a bad thing: people are working hard in your company to get things done.
Elsewhere in your organization, an expert needs just that one visualization your BI tool lacks. Or that data source. Or that algorithm. Soon enough, there will be one more tool for you to manage when you find out about it, belatedly. It will be a little more specialized and gaining a foothold. And nor is this a bad scenario; we're just seeing more people working to do the right thing.
Your mergers and acquisitions will bring new architectures and applications to live alongside your own. It can take years to merge and rationalize across those orgs, and meanwhile, daily work still has to go on. And in the BI market, companies and technologies come forth, grow, stagnate and are bought and sold. So where does that leave your plans?
This is just the way of the business.
A data hub: future-proofed BI architecture
You can, however, design an architecture to cope with change from the first day. To do so, your aim should be to insulate systems from breaking changes while enabling simple integration and rapid development of new designs.
Traditionally, data warehouses held out some of these possibilities. Users might access the underlying data models from a wide variety of tools, even the latest shiny BI visualizations. Architects might add new sources to the data warehouse or even new subject areas following a merger. Administrators maintain the authority of the data by strict quality controls.
Yet a data warehouse such as this seems unlikely to be your answer either. No doubt the architecture can live on, for certain subject areas where absolute control of the process is more important than flexibility. Think (if you must) of tax reporting, charts of accounts and legal structures.
For your other myriad use cases such as marketing, supply chain, sales analysis and production, a data hub will more likely meet your needs.
A hub integrates with just as many sources and clients as a data warehouse. In the late hours, your users may reach for Excel rather the BI platform, but they source data from your hub as easily as from any other Excel table.
The hub, through automated development, makes possible both quality control and rapid iterations of design. You can keep up with new requirements and still maintain consistency.
Moreover, a hub insulates your architecture from organizational and technical changes over which you have no control. Administrators can merge in the sources of newly acquired businesses without disrupting existing models. Architects might swap out the entire front end of a Business Intelligence system and still source their data consistently from the hub.
Many BI tools have data servers that aim to achieve some hub-like functions of consistency and integration, but they are highly proprietary and best used as optimized caches for the particular needs of that tool: exploring, drilling down, serving mobile apps and so on. For such purposes, a specialized data server may be unmatched. Today, when you may have only one front end, that client-focussed data layer may sound like an attractive option. But tomorrow, next quarter, next year? Who knows how many tools will be in use or where your strategy will turn?
Our businesses are unpredictable, sometimes maddeningly so. That's much of our work and most of our fun.
In my experience, whether working on SQL Server, PowerPivot or Qlik, data architectures work best when they meet the real world needs of an evolving environment and a changing toolset. I like hubs for that reason, and with automated development, I like them even more.