With the intelligent execution engine of TX DWA, you get performance levels almost impossible to achieve using traditional, hand-coded approaches. Standard performance enhancing techniques such as source or target based incremental load can be enabled with the flick of a switch, enabling you to concentrate on more interesting parts of building a data warehouse.
Managed Thread Execution
TX DWA utilizes an automated method of project execution that enables much faster execution than it is possible with traditionally developed approaches. In a typical setup, execution time can be improved 20%-60%, often halving the execution time. We call it Managed thread execution. Managed, because it optimizes the execution order while taking care of dependencies between objects, and thread because it utilizes multiple threads to speed up execution.
Managed thread execution lets you execute projects with as many threads as you like. This enables you to fully utilize the hardware TX DWA resides on, moving any bottle necks from software to hardware.
TX DWA optimizes the execution order by executing the biggest tasks as early as possible while taking dependencies between tables, cubes and other objects into account. Execution times for the individual objects are logged and TX DWA uses this information to decide what tasks takes the longest and should be executed first. In this way, the execution order is not only optimized once in the beginning, as you might do when coding “by hand”, but each time the project is executed.
When you halve execution times compared to traditional single-threaded un-managed execution, it opens up a world of opportunities. For instance, it enables multiple daily updates of even large business intelligence solutions, ensuring that you always have an up-to-date basis for decision-making.
Per default, the intelligent execution engine tries to complete execution as quickly as possible, which makes sense most of the time. However, you might sometimes have other considerations. For instance, you might only have a few hours each night to fetch data from an otherwise busy server. Using the prioritization feature, you can give priority to certain tables and sources to make sure they are processed as early as possible. You can choose to prioritize all steps related to a specific table or just, for example, the data transfer. You can also choose to block all other execution tasks until a certain step is completed.
Failure is a fact of life in data warehousing and solutions based on TX DWA fail from time to time as well. Sometimes it is a error in the project, other times it is a source system that is unresponsive. In TX DWA, failures and warnings are logged and we have also built in features to handle these situations.
You probably have some source systems in your solution that are less than critical for your reporting. You can configure TX DWA so that the entire execution does not stop just because TX DWA cannot reach these noncritical systems. Instead, you can choose to keep the newest data from the system in question until fresh data can be fetched.
When a critical error do occur and a three-hour execution fails halfway through, a large amount of data has already been processed before the execution stopped. In TX DWA, you can fix the error and resume the execution from where it stopped, saving sometimes hours of waiting.
TX DWA supports both source and target based incremental load. The point, in both cases, is to minimize the need for loading and transforming the same data over and over, significantly speeding up the overall ETL process.
Source-Based Incremental load takes advantage of situations where TX DWA can rely on the source system to identify new or modified records. TX DWA then only loads these records from the source system, skipping unchanged records.
Source-Based Incremental Load can be combined with Target-based Incremental Load for even better performance. The target-based approach controls how data is inserted, updated, and deleted between raw and valid table instances, which improves load times on very large tables.
Unlike the source-based approach, Target-based Incremental Load works even when the source system cannot tell which records are new or modified.
When you use Target-based Incremental Load, one or more fields that uniquely identify a record are hashed together into a new system field that is used to compare incoming and existing records. TX DWA the uses a second hashed value, based on a selection of fields chosen by the user, to decide which records to update in the valid table. This minimizes the number of updates, thus improving the load performance.
Interested in TimeXtender Software?
Interested in Partnering with TimeXtender?
VP, Partner Development & Sales
Phone: +1 877 572 3477
VP, Partner Development & Sales
Phone: +45 8620 5700
Director, Partner Development & Sales
Phone: +46 701 46 85 98