Automated migration from Netezza to AWS for a Fortune 500 mortgage lender

Lowered operational cost and improved data visibility across the enterprise

Challenge

A US-based Fortune 500 mortgage lender wanted to move their Netezza-based enterprise data warehouse to a modern data lake architecture. Additionally, they wanted to reduce licensing and maintenance costs, meet data consumption and business SLAs, and achieve faster time-to-market.

They were looking to rearchitect and migrate workloads from Netezza to an AWS-based enterprise data lake with the following capabilities:

• Scalable services and customized applications in line with current and future business needs
• Ability to move reporting/analytical consumption loads to a scalable AWS vending layer
• Robust disaster recovery
• On-demand cloud-native ETL (Extract, Transform, Load)

The client wanted to leverage cloud-native ‘as-a-service’ offerings to reduce management complexities and switch to a more economical pay-as-you-go pricing model.

Solution

To meet the client’s business needs, the Impetus team designed an enterprise-scale hybrid cloud solution and leveraged LeapLogic to automatically translate Netezza logic to the target environment.

Solution highlights:

• Identified expensive and poorly performing data processing entities through ML-based assessment
• Migrated historical data and developed a reliable mechanism for incremental updates to S3
• Migrated 100+ NZSQL files with approximately 20 queries and 5 Informatica ETL workflows
• Created structured data lake zones on S3 (landing, preparation, and insights) for quality check and data augmentation for downstream consumers
• Automatically translated consumption queries (i.e., SQL, stored procedures, Shell, and Scheduler) into Spark SQL/Hive QL
• Ingested data from S3 to Redshift for reporting and analytics
• Performed SCD (Slowly Changing Dimensions) Type 1 and Type 2 for slowly changing dimensions
• Replicated the Netezza snapshotting process on AWS
• Used Autosys and Oozie as on-premises/cloud orchestrators
• Used AWS Lambda as an event-driven notification engine
• Used Amazon EMR to process vast amounts of data across scalable Amazon EC2 instances
• Used Amazon RDS (Relationship Database Service) to store metadata and workflow information
• Evaluated and performed tool benchmarking for consumption queries on the cloud
• Provided on-site support and services to ensure seamless migration

Impact

LeapLogic migrated and transformed workloads from Netezza to AWS, enabling a Fortune 500 mortgage leader to realize the following benefits:

• Lowered operational expenses by freeing up capacity in Netezza
• 50% time and cost savings compared to manual migration
• Improved data availability across the enterprise, enabling innovation