Case Study

Cost optimization by transforming Teradata batch and ad hoc workloads to Hadoop


25% cost reduction using automation


Challenge

An American multinational retail chain wanted to optimize and reduce the current cost of their legacy data warehouse and build a data lake for faster ETL processing and use of advanced analytics while reducing time to analytics. To achieve this, the client wanted to transform batch applications and ad-hoc queries that consume expensive Teradata cycles to the Hadoop/Hive environment.

  • Batch transformation includes ingestion of dependent datasets from the source data store, building batch jobs, uploading transformed data to target systems, performing data validation, warranty support, and production hand-off.
  • Ad-hoc transformation includes establishing access patterns, ingestion of dependent datasets from the source data store, query conversion along with conversion of Teradata specific operators to the target systems, and user-training assistance.

 

Improved performance by 30% using automation

 

Solution

LeapLogic, an Impetus product for automated workload transformation enabled one-time migration of historical data from Teradata to DB2. LeapLogic features an automated utility that converts BTEQ and SQL transformation scripts into equivalent Spark QL/ HiveQL and executes them on the Hadoop/Hive environment. It also allows users to run a set of data validation checks. Finally, the post-processed analytical data can be loaded back to the source enterprise data warehouse for reporting and access.

 

Improved scalability by transforming workloads to a modern stack

 

Further, a combination of LeapLogic and IBM DataStage were used for incremental data transformation from the source data store to the modern data platform. Processed results can also be updated back to Teradata or other source systems.

A high-level functional component architecture of the implemented solution is given below:

Client’s data fabric balancing solution architecture
Client’s data fabric balancing solution architecture

 

Impact

Our comprehensive solution helped the client move from Teradata and realize the following benefits:

  • Reduced cost by 25%
  • Improved performance by 30% using automation
  • Improved scalability by transforming workloads to a modern stack
  • Improved user experience and reduced business risk