IAG Finds Equilibrium in Data Mesh Endeavor
We independently review everything we recommend. When you buy through our links, we may earn a commission which is paid directly to our Australia-based writers, editors, and support staff. Thank you for your support!
IAG’s Intermediate Approach to Implementing Data Mesh
Quick Read
- IAG has been a year into the implementation of a data mesh architecture.
- The organization is utilizing Google Cloud and Confluent solutions.
- The utilization of real-time data streaming is expanding within IAG.
- Google Cloud Platform (GCP) serves as the cornerstone of IAG’s strategic data platform.
- Confluent Cloud enables internal use cases for real-time streaming.
IAG’s Path to Implementing a Data Mesh
IAG is one year into adopting a data mesh architecture to further enhance the importance of data in its group operations and transformation initiatives. Burak Hoban, the executive manager of data platforms – data and risk, provided insights during the Confluent Data in Motion Tour 2024 event in Melbourne, expressing that IAG seeks to strike a balance in its data mesh journey.
“We’ve already put that into practice, and we’re a year into the process,” Hoban mentioned. “My responsibility is to help expedite it.” Two key components of IAG’s data mesh include its strategic data and analytics platform, which utilizes Google Cloud services like BigQuery, alongside Confluent Cloud and connectors.”
Fundamental Tenets of Data Mesh
Data meshes are constructed based on several fundamental principles. These encompass treating data as a product, integrating data into a central repository while maintaining ownership and curation by “the domain team most acquainted” with the dataset, enabling self-service data access, and implementing universal standards for all data in use. Hoban noted that IAG had invested several years in establishing the essential components for a data mesh.
The use of real-time data is increasing.
IAG is beginning to identify more use cases for real-time streaming data throughout the organization. Hoban mentioned that the potential of real-time streaming and business eventing was acknowledged as early as 2017-18. Although some use cases did eventually surface, Hoban admitted that they were “slightly ahead of their time” overall.
Initial streaming implementations were internally constructed and overseen using Apache Kafka. However, as usage escalated, the insurer transitioned to a managed Kafka service via Confluent Cloud. This shift enabled Hoban and his team to collaborate more effectively with internal development and engineering teams, thereby boosting Kafka adoption.
“Kafka is beginning to take on a crucial role in our strategic platforms for the future,” Hoban remarked. He also hinted at a greater utilization of pre-built connectors provided by Confluent to access and stream data from different source systems.
Fully Committed to Google Cloud Platform (GCP)
At a Google Cloud Summit in Sydney in May, IAG’s Executive General Manager of Data, Risk, and Resilience, David Abrahams, shared additional insights about the insurer’s data platform built on GCP. Previously, the insurer had only mentioned its use of GCP in passing and had not discussed its architecture or decision-making process in detail.
“We possess a substantial amount of data that is crucial to our operations; however, due to our legacy systems, this data has become fairly fragmented and isolated,” Abrahams stated. IAG established Google Cloud as the cornerstone of its “strategic data and analytics platform.” This platform now utilizes Google Data Platform for advanced analytics, employing Vertex AI and machine learning on BigQuery.
The outcomes from this platform encompass the capability to craft more personalized experiences for customers and enable business teams to independently launch new models into production without requiring extra technical assistance.
Summary
IAG’s adoption of a data mesh architecture demonstrates its dedication to updating its data operations. Utilizing technology from Google Cloud and Confluent, IAG has established a framework that enables real-time streaming applications and self-service data access. This method improves customer experiences and grants internal teams increased independence in their data processes.
Q&A
A: Could you explain what a data mesh is?
A:
A data mesh is a design strategy that decentralizes data governance by considering data as a product. This allows domain-specific teams to manage and refine their datasets while following common guidelines.
Why did IAG opt for Google Cloud Platform?
A:
IAG selected Google Cloud Platform for its ability to provide a scalable and high-performance solution that could effectively handle its extensive, fragmented, and isolated data. This choice facilitates enhanced analytics and machine learning functionalities.
What function does Confluent Kafka serve in IAG’s strategic plan?
A:
Confluent Kafka enables IAG’s real-time streaming applications. It assists in synchronizing data across business tools, supports customer and policy migration processes, manages payment notifications, and helps develop real-time data products.
How has utilizing a managed Kafka model proven advantageous for IAG?
A:
By utilizing the managed Kafka model via Confluent Cloud, IAG has been able to free up its internal resources. This shift has enabled the team to concentrate on development and engineering activities, thereby increasing Kafka adoption throughout the organization.
Q: What are the advantages of IAG’s strategic data platform?
A:
IAG’s strategic data platform enhances customer experiences through greater personalization, allows business teams to independently deploy new models, and facilitates advanced analytics using Vertex AI and machine learning on BigQuery.
Q: What effect does real-time streaming have on IAG’s operations?
A:
Real-time streaming enables IAG to manage essential functions like generating pricing quotes or issuing policies effectively. Any downtime or problems with Kafka can severely affect these activities, underscoring its crucial importance.
