3 min read
Growth of the amount of data, data sources and analysis possibilities ensure that there are also multiple possibilities for storing and processing data. All kinds of concepts are on the table — such as Data Lake, Data Hub, Data Warehouse and Data Platform. In practice, there is a great deal of ambiguity about what these concepts actually entail; so how do they differ from each other? This blog provides more clarity on the meaning of these terms.
Data Warehouse
A Data Warehouse (DWH) consists of the integrated storage of information with the aim of feeding business decisions and analyses. With a data warehouse you lay the foundation for Business Intelligence (BI) and Analytics. The data that is in a data warehouse often comes from different sources inside, or outside, the organization.
Because a data warehouse brings together large amounts of data from different data sources (HRM, DMS, ERP), valuable insights can be gained. A data warehouse saves time for companies that collect data on a large scale and ensures uniformity of definition of business information. When you want an integral understanding of your business, a data warehouse is important.
Data Lake
A data lake is a storage location in which large amounts of raw data are stored in its original structure. In terms of content, a Data Lake can contain structured and unstructured data. The data structure of individual files and how they should be accessed is not known until the data is used. It is important that a data lake is not seen as a replacement storage system, but as a place where analysis and research can be done with unprecedented freedom due to the relatively low cost of storage and the ease of scaling up. Data lakes are generally a good basis for reporting, visualizations, advanced analytics and machine learning.
(Un)Structured data?
- Structured: data from databases, CSV, JSON, etc.
- Unstructured: email, PDF, documents, video, audio, binary files
Data Hub
A data hub does not store data itself, but takes care of the flow of data between source systems and target systems and users. With a data hub you indicate exactly what needs to be done with the data. You can link certain information from sensors to an automated order system, for example. The power of the source system is used as much as possible to ensure optimal performance. Often, a data hub takes the form of a hub-and-spoke architecture where systems can distribute data through the Data Hub, rather than through point-to-point integration where every system is connected to any other system with which data needs to be shared.
In addition, a data hub provides organizations with insight to be able to interpret data properly. Because if you understand what you're looking at, it becomes easier to ensure the accuracy of data or adjust it where necessary. You can literally see how datasets are constructed up to the column and row level. Moreover, you can comply with laws and regulations, because you know exactly who has access to what data and where data is stored. The data in a data hub is not necessarily integrated and can contain different levels of detail side by side as opposed to a Data Warehouse. Set against a data lake, a data hub can offer data in different formats. Where data warehouses and data lakes are endpoints for data, a data hub is a node through which data flows.
Data Platform
A data platform is an integrated solution that combines the functionalities of data lake, data warehouse, data hub and elements of a Business Intelligence (BI) platform. Without a data platform, a separate tool or set of tools is usually used for each aspect. This creates a complex environment where many tools need to be managed to make data flow from source to end user. A data platform centralizes these solutions in one place and delivers a product that is a lot more manageable.
Differences
Schema on read/write?
- Schema-on-read: Data is stored unchanged
- Schema-on-write: data is transformed and stored in a predefined structure
Conclusion
The significant increase in data sources and volume, and the different data needs of different users, pose real challenges for BI and IT departments engaged in analytics, artificial intelligence (AI) and reporting. Organizations use a variety of tools to process and manage data. There's another way.
The Timextender Data Platform provides a unified data infrastructure for cloud, hybrid, and on-premises environments. This allows you to connect to any data source and ingest, prepare, enrich, and document data for analytics and AI purposes — with automatic lineage and documentation generated throughout.
Timextender automates the repetitive, time-consuming work involved in building a traditional data platform. With Timextender, you can build an integrated data solution that delivers AI-ready data up to 10 times faster, while reducing management overhead by up to 80%.