Traditional approaches to data warehousing have changed--has your data warehouse?
“Old habits die hard,” says Wayne Eckerson, a thought leader in the business intelligence and analytics field since the early 90s. “Too many companies rely on traditional hand-coded and labor intensive approaches to building a Data Warehouse.”
If you're responsible for building or maintaining a data warehouse, that probably rings true for you. And in this age of automation, in which robots are used for everything from milking cows to gathering customer data on a phone call, why is it that it's taken so long for automation to be incorporated into the data warehouse?
The data warehousing process historically has gone something like this: gather requirements from business users, design a data model to support those requirements, locate the data sources, and load the data into a star schema and develop Business Intelligence (BI) objects. Perfectly logical, right? Until the business team realizes that what was considered ideal during the requirements process isn’t so great now. Or that what would be much better this month is to see the rate of fulfillment instead of rate of conversion--but the fulfillment data hadn't earlier been deemed worthy of importing into the warehouse.
The traditional, static data warehouse is as passé as Air Jordans, backwards baseball caps and the Macarena.
A key tenet: all data is valuable
Today's modern data warehouse requires a level of flexibility to keep up with changing business needs. That's why we've dubbed the modern system the Data Discovery Hub, which is distinguished from traditional data warehouses by its agility, simplicity and flexibility, all of which are enabled by simple automation.
A key tenet of the Data Discovery Hub is this: import all your available data. Don't waste time and energy deciding which data should be brought in and which won't be useful.
Think of it this way: imagine you're going to move to a new house. Conventional wisdom requires that you painstakingly sort through every object you own before moving and anticipate whether or not you'll need it in your new home. This process is not only time-consuming but also runs the risk of leaving behind items that could be useful but weren't determined to be so at the time. And then you have to either rush to replace those items or simply do without.
A data discovery hub works more like this: every item you own is packed up into neatly labeled boxes and automatically sorted for later use. If you planned to throw a monthly pool party but end up deciding to throw a fondue party instead (even though you never thought you would do that again), your fondue pot and skewers are automatically dusted off and presented to you in pristine condition. Nothing is excluded; all data is assumed to be valuable.
This is the major difference between a Discovery Hub approach and the classic approach of building a star schema where only select data can be discovered. By rejecting status quo and importing all available data, your data warehouse is not only more complete, but the process of creating it is much simpler, more elegant and fully governed.
Additionally, business owners are often grateful when IT takes the Discovery Hub approach, since the data is more agile, accessible and actionable--three things that business users appreciate.
Why automation is a well-kept secret
So why aren’t more companies shifting to an automated approach to data discovery? According to Eckerson, “Data warehouse automation is a really well-kept secret.”
And it's understandable why awareness and adoption are slow to spread: many companies have multi-million dollar investments in traditional data warehouse tools entrenched in the status quo, and reimagining the data warehouse may feel like an overwhelming task. Developers may fear that automation tools will put them out of work. And consultancies may derive the majority of their income through hand coding and so may not be eager to upset their current business models.
However, most of those fears are unfounded. While adding automation to a traditional data warehouse may seem like a cumbersome project, most automation can be up and running in a matter of days.
As for developers, data warehouse automation frees them to focus on mission-critical tasks. And consultancies tend to get the best word-of-mouth by amazing their customers and delivering instant results--and most organizations that add automation see results within days or weeks rather than months or years!