Organizations want to leverage heaps of data generated from business processes in a self-serve mode to promote data driven decisions. However, they are facing challenges to manage the underlying data platform. While the current Data Platforms do provide ability to generate insights from data, they still struggle with the issues of siloed components, centralised governance, and lack of Data trust to accomplish self-service. Data Mesh offers a solution to these problems with its principles around Domain Driven Ownership, Data as a Product, Self-Serve Data Infrastructure Platform and Federated Computational Governance. In this article, we will focus on creating Data as a Products. One can know more about Data Mesh here
Data as a Product v/s Data Product –Data Product term traditionally has been used to define a product that facilitates an end goal through use of Data; Data Mesh proposes to offer data assets as a product and hence uses the term ‘Data as a Product’ to apply product thinking on Data Domain to offer Data Assets as a Product. Hence in the context of Data Mesh, we use Data Product as short of Data as a Product.
There are 3 kinds of Data Products –
1. Source Aligned — These reflect the business facts as generated by operational systems.
2. Aggregate Data Products — These are aggregate of multiple upstream domains.
3. Consumer Aligned — These are created towards fulfilling one or multiple specific use cases.
For implementing a complete Data Mesh architecture, while individual Data Domains have liberty to select the technology to implement Data Products, there are certain architectural components that are decided at the Data Mesh level as mentioned below –
1. Platform for Data Marketplace; Data Governance and Security.
2. Integration Layer for sharing of raw data.
3. Monitoring platform for various performance metrices.
Central Team would implement these platforms and provide self-service infra as a code to create Data Products. Data Products align to a Domain and a Domain can offer multiple data products. Domain team is responsible for selecting the implementation technology and supporting the Data Products that it offers. The Data Product defines its own SLOs and adheres to those.
To capture the offering of the Data Product, we can use Data Product Canvas. Just like a Business Model Canvas provides snapshot of Business in one page, Data Product Canvas provides an overview of Data Product in a single sheet. It clearly articulates what is the value preposition of Data Product, what its sources are and what would be the consumers along with potential monetization and governance part. Data Product canvas serves as a practical tool for implementing the Data Product.
These are the main sections of a Data Product Canvas –
a. Data Sources — This section captures what all sources will be feeding data into Data Product. What would be the frequency, type, and mode for the data feeds.
b. Transformations — This section captures key transformations done on the source data to create the Data Product.
c. Data Domain — This section captures what domain does the data product belong to.
a. Value Preposition — This section captures what is the core value preposition of the Data Product and answers the key question of Why Data Product needs to be there in the first place.
b. Data Product Description — This section describes the Data Product in terms of owner and usage.
c. Channels — This section describes how the Data Product will be consumed.
3. Consumers [Consumer Segments] — This section talks about different consumers for the Data Product offering.
4. Governance and Security — This section captures how the data product will be governed and secured.
5. Monetization — This section describes the potential to monetise the Data Product.
Example — Nile is an e-commerce company selling products online. Its also provides a B2C marketplace for various vendors to sell their products over its platform. We are proposing to create a Data Product used to analyze the orders across customer segments , geographies.
External Reference — https://en.wikipedia.org/wiki/Business_Model_Canvas
Architect — Data, Cloud , AI/ML