Automation-driven holistic modernization of the Oracle Data Suite to a cloud-native stack - LeapLogic
Blog
12 Jan 2023

Automation-driven holistic modernization of the Oracle Data Suite to a cloud-native stack

In 2023, approximately 120 ZB of data was created worldwide. By 2025, people will create more than 181 ZB of data.

Source – Statista

 

To harness the full potential of this massive volume of data, enterprises leverage data suites like Oracle. Being easy to deploy and manage, the data suite offers a full spectrum of enterprise data warehousing and ETL capabilities. This includes data ingestion, data integration, business analytics, IoT, etc., with tools such as Exadata, ODI, etc., and business intelligence systems such as OBIEE.

However, legacy systems like Oracle come with their own set of challenges, such as hefty infrastructure and maintenance costs, data silos, lack of auto-scalability, data censorship, handling of unstructured data, prolonged processing time, etc., for which it needs to be integrated with different third-party applications, tools, or cloud business processes. These complexities can deter your transformation efforts and push your company into a business rut.

To combat the challenges posed by legacy systems, enterprises need to migrate their data estate from legacy sources like Oracle to a modern cloud-native stack. But what are the aspects that you need to consider for risk-free modernization?

Modernization is much more than ‘lift-and-shift’ or complete re-engineering. It needs to strike an intelligent balance between both and ensure seamless migration of workloads, business logic, and codes.

The blog explores various nuances and critical considerations for migrating different types of legacy workloads, such as procedural code, ETL scripts, reporting scripts, etc., to cloud environments.

Key considerations for Oracle data warehouse migration – Schema, data, PL/SQL, and sub-query

Schema and data

Migration is all about reusing existing workloads and applies to both schema and the underlying data. Schema and data migrations should be performed efficiently, as even the slightest error can result in disastrous consequences such as data loss and schema inconsistencies.

As part of your schema migration plan, data type mapping between the Oracle data warehouse and the target cloud platform should be an important consideration. Furthermore, a successful schema migration plan should start by answering the following questions:

  • How will the constraints and important keys be migrated or treated in the target cloud data warehouse?
  • What kind of schema optimization strategy and target-native best practices may be required?
  • How should the existing technical debt be handled at the schema level to optimize CPU/IO capacity, cache hit ratio, etc.?

Similarly, for data migration, some of the critical aspects that require focus are:

  • Incremental data migration
  • Ensuring data security while data is in motion
  • How CDC will be implemented on the target platform
  • How will SCD be handled and indexes be managed
  • Whether data needs to be replicated to multiple targets
  • Should data be consolidated from multiple sources?

Owing to these considerations, an end-to-end automated migration solution is imperative as it mitigates any risk and ensures a smooth transition to the target environment.

PL/SQL

Oracle’s PL/SQL combines the power of SQL with procedural codes and helps to control the flow of execution through conditional branching and iterative processing. Procedural code may contain cursors making it easy to manipulate the contents of database tables. It may also be written to carry out bulk processing operations, etc.

Let us take an example of cursors from the migration perspective. When dealing with large amounts of data, cursor loop statements may fetch and process data indefinitely, making the process more time-consuming and affecting the SLA performance. Such situations can be easily resolved by converting the Oracle PL/SQL cursors to the cloud-native equivalent using a set-based approach. This approach creates a temporary table and directly updates it instead of a cursor.

Sub-query

Like PL/SQL, sub-queries must be converted to a QUALIFY clause to overcome performance inefficiencies. For instance, look at the example below of a converted sub-query.

 

select * from (
select salesid, max(qty) as max_qty,
row_number() over (partition by sellerid order by qty asc) as row1
from sales
group by 1
) abc
where row1 <= 10;

select salesid, max(qty) as max_qty
from sales
group by 1
QUALIFY row_number() over (partition by sellerid order by qty asc) <= 10;

 

Key considerations for ETL migration – Oracle Data Integrator (ODI)

ODI ETL is complex compared to other ETL tools. As ODI queries are not clearly defined as SQL queries, it is difficult to identify and convert them. The queries are segregated into tag lines that contain multiple attributes. For example, it is difficult to identify the different components of Join queries since multiple queries fall under the same tag.

Additionally, the ETL processing cost for vast amounts of data is substantial. The ODI tool also encounters difficulties when used on large data sets and often fails to handle real-time data integration. Furthermore, while some features of ODI are intuitive, others are not, making troubleshooting complex.

The tool also demands an in-depth understanding of Oracle products and systems to understand and use them effectively. Additionally, the documentation generation process takes longer than usual and proves challenging for technical users to grasp. Furthermore, connections to external data sources are subject to timeouts after a specific period, leading to disruptions in data ingestion and transfer. These complexities contribute to the scarcity of qualified individuals in the market, resulting in a skillset gap.

Key considerations for reporting workloads migration – Oracle Business Intelligence Enterprise Edition (OBIEE)

You can fully leverage a cloud-native stack only after moving different kinds of reporting workloads to the cloud, which eventually completes the migration process. The reporting workloads may include hundreds of dashboards, reports, and related entities such as publishers, subject areas, models, agents, presentation tables, logical tables, physical tables, and more.

The reporting dashboards and other workloads, such as ETL scripts, orchestration logic, stored procedures, DML statements, data, schema, etc., become your complete set of workloads for migration. Within these lists of workloads, it’s imperative to identify dead code, technical debt, used and unused workloads (queries, tables, procedures, macros, views, etc.), poor-performing and resource-constraining workloads that need to be optimized, duplicate code, etc.

Automating the transformation lifecycle: A methodical cover-to-cover approach

A manual approach to workload modernization can be time-intensive, error-prone, and lead to unwanted business challenges. However, by leveraging the power of automation, you can save time, minimize risk, and ensure business continuity.

LeapLogic, Impetus’ cloud accelerator, drives end-to-end transformation of your Oracle data estate workloads to the modern cloud data architecture, with up to 95% automation. It fast-tracks and simplifies the end-to-end transformation of your Oracle data warehouse, ODI, and OBIEE workloads to a cloud-native stack in four easy steps – assessment, transformation, validation, and operationalization.

Assessment

  • Integrated assessment for several workload types such as ETL, data warehouse, BI/reporting, orchestration scripts, etc.
  • Comprehensive inventory listing of source
  • Workload prioritization as per the business use case, SLAs, cost, etc.
  • Advanced blueprint of the target architecture
  • Identification of optimization opportunities for schema, code, and orchestration
  • Prescriptive recommendations for effective and automated migration
  • Extensive interdependency analysis between various workloads in the form of data and process lineage
  • Phased migration plan with timelines and cost estimates along with a detailed implementation strategy

Transformation

  • Intelligent transformation engine supporting a variety of workloads and source formats, including DML, procedural code, orchestration logic, ODI ETL scripts, OBIEE BI reports, and proprietary scripts such as Oracle Spool, etc.
  • Cloud-native conversion, packaging, and orchestration as production-ready jobs
  • Deployment of an executable package on the target cloud environment for the end-to-end consummation of the migration lifecycle
  • Delivers a verified, executable package with performance SLAs met on the target side
  • Preservation as well as optimization of core business logic, including procedural code, orchestration logic, ETL scripts, and more
  • Extensible, repeatable, and verifiable transformation methodology

Validation

  • Pipeline-based automated validation on diverse datasets
  • Auto-generation of reconciliation scripts for faster and automated validation and certification
  • Automated SQL/query and data-level validation for higher coverage
  • Cell-to-cell validation reports, i.e., to the minutest level
  • Data type and entity-level matching

Operationalization

  • Target cloud-native orchestration for end-to-end smooth transitioning into production
  • Infrastructure-as-Code (IaC) for automation on the operationalization front
  • Provisioning of the required cloud services and setting up policies such as auto-scaling etc.
  • Automated DevOps, including CI/CD, etc.
  • Target platform stabilization with parallel runs
  • Setting up the right level of security protocols and policies as per the organization’s specific requirements
  • Controlled price-performance ratio on the target
  • Smooth cut-over and legacy footprint decommissioning
  • Documentation, training, and handholding via runbooks, training sessions, etc.

Empowering transformative Oracle Data Suite migration for future-ready success

LeapLogic is the catalyst for successful transformation, aiding numerous Fortune 500 enterprises in seamlessly transitioning their legacy Oracle data suite to a preferred cloud environment. With LeapLogic, the modernization process becomes swift and efficient, enabling enterprises to:

  • Experience controlled costs and reduced migration expenses
  • Safeguard critical business logic, preserving the integrity of their data suite.
  • Foster team collaboration, empowering self-service and boosting efficiency.
  • Achieve streamlined migration, promoting operational excellence and reducing uncertainties.
  • Get the most from investment with end-to-end automation and resource efficiency.
  • Minimize time-to-value in the modernization journey, thereby opening doors to innovation and business growth.
  • Benefit from auto-scaling, aligning resources seamlessly with workload requirements.

Leap into the future confidently, knowing your Oracle data suite is in capable hands with LeapLogic.

Gurvinder Arora
Author
Gurvinder Arora
Product Marketing Manager