NAB Overhauls Data Pipelines for Advanced Ada Platform

Quick Read

• NAB is standardizing its data pipelines for the Ada platform using Spark Declarative Pipelines (SDP).
• The aim of this initiative is to lower expenses and launch new recovery functionalities.
• The Ada platform functions on Databricks that is hosted on AWS.
• Standardization began with the bronze layer and is now progressing to the silver and gold layers.
• SDP provides a declarative method for constructing data pipelines, enhancing efficiency.
• NAB anticipates a 15% decrease in operational costs upon complete implementation of SDP.
• A new “rewind and replay” capability aids in restoring data to a consistent state.

Transforming Data Pipelines with SDP

The National Australia Bank (NAB) is revamping its data operations by standardizing its data pipelines for the Ada platform with Spark Declarative Pipelines (SDP). This transition is set to not only cut operational costs but also boost the recovery capabilities of the platform.

The Journey with Ada

Initiated as a major milestone in NAB’s data strategy, Ada signifies the second phase of the bank’s data transformation journey. Since launching in 2022, the platform has optimized NAB’s data framework by merging multiple systems, such as the older NAB Data Hub and a legacy Teradata environment, into a more effective infrastructure based on Databricks on AWS.

Implementing Medallion Architecture

A pivotal element of Ada’s design is the adoption of the medallion model, a widely used structure among Databricks users. This model classifies data into three layers: bronze, silver, and gold. At first, the intake of raw data into the bronze layer was standardized using SDP, paving the way for further developments at the silver and gold levels.

Embracing Declarative Pipelines

The transition to SDP brings a declarative method to creating data pipelines. Instead of writing code for each extract, transform, and load (ETL) step, engineers outline the required data outcomes. Databricks’ Lakeflow platform then handles the orchestration, state management, and incremental processing needed to realize these results, simplifying the entire workflow.

Progress and Future Goals

Progress is already visible in the silver layer, where new pipelines are constructed with SDP, while existing ones are being modernized. NAB’s long-range objective is to fully implement SDP across all layers, ultimately establishing a benchmark for data pipeline practices in the financial industry.

Cost Savings and New Features

The shift to SDP is projected to create considerable cost advantages, with NAB estimating a 15% decline in operational expenses when all custom Spark jobs are retired. Furthermore, NAB’s engagement in the private preview of the innovative “rewind and replay” feature presents a novel solution for data recovery, allowing pipelines to revert to a previously consistent state when needed.

Summary

NAB’s effort to modernize its data pipelines utilizing SDP is a strategic approach aimed at improving efficiency and cutting costs. By adopting a declarative model and utilizing the medallion architecture, NAB is establishing new benchmarks in data management within the banking industry.

Question and Answer