Automated migration from Netezza to AWS for a Fortune 500 mortgage lender
Lowered operational cost and improved data visibility across the enterprise
Challenge
A US-based Fortune 500 mortgage lender wanted to move their Netezza-based enterprise data warehouse to a modern data lake architecture. Additionally, they wanted to reduce licensing and maintenance costs, meet data consumption and business SLAs, and achieve faster time-to-market.
They were looking to rearchitect and migrate workloads from Netezza to an AWS-based enterprise data lake with the following capabilities:
- Scalable services and customized applications in line with current and future business needs
- Ability to move reporting/analytical consumption loads to a scalable AWS vending layer
- Robust disaster recovery
- On-demand cloud-native ETL (Extract, Transform, Load)
The client wanted to leverage cloud-native ‘as-a-service’ offerings to reduce management complexities and switch to a more economical pay-as-you-go pricing model.
Solution
To meet the client’s business needs, the Impetus team designed an enterprise-scale hybrid cloud solution and leveraged LeapLogic to automatically translate Netezza logic to the target environment.
Solution highlights:
- Identified expensive and poorly performing data processing entities through Intelligent assessment
- Migrated historical data and developed a reliable mechanism for incremental updates to S3
- Migrated 100+ NZSQL files with approximately 20 queries and 5 Informatica ETL workflows
- Created structured data lake zones on S3 (landing, preparation, and insights) for quality check and data augmentation for downstream consumers
- Automatically translated consumption queries (i.e., SQL, stored procedures, Shell, and Scheduler) into Spark SQL/Hive QL
- Ingested data from S3 to Redshift for reporting and analytics
- Performed SCD (Slowly Changing Dimensions) Type 1 and Type 2 for slowly changing dimensions
- Replicated the Netezza snapshotting process on AWS
- Used Autosys and Oozie as on-premises/cloud orchestrators
- Used AWS Lambda as an event-driven notification engine
- Used Amazon EMR to process vast amounts of data across scalable Amazon EC2 instances
- Used Amazon RDS (Relationship Database Service) to store metadata and workflow information
- Evaluated and performed tool benchmarking for consumption queries on the cloud
- Provided on-site support and services to ensure seamless migration
Impact
LeapLogic migrated and transformed workloads from Netezza to AWS, enabling a Fortune 500 mortgage leader to realize the following benefits:
- Lowered operational expenses by freeing up capacity in Netezza
- 50% time and cost savings compared to manual migration
- Improved data availability across the enterprise, enabling innovation
You may also be interested in…