Data Powered Applications
Organizations that effectively leverage data insights for decision-making will be at the forefront of market leadership. While many organizations currently utilize data to some extent, the next phase of growth will be driven by those that integrate data-driven insights into every aspect of their decision-making processes. The era of senior managers relying on monthly or quarterly business intelligence reports for executive decisions is over. The current imperative is to provide operational users with real-time insights to inform their daily activities.
While this concept is straightforward to articulate, it necessitates a fundamental shift in application development methodologies. There will always be a distinction between analytics systems and transactional systems due to their inherently different storage and retrieval mechanisms. Transactional systems, utilized by operations, rely on relational databases optimized for row-wise storage and retrieval of information. Conversely, Business Intelligence systems depend on columnar analytics databases designed to efficiently process large volumes of data. Additionally, there are various other databases such as graph, document, vector, and NoSQL, each optimized for specific data storage and retrieval formats. Given these optimizations, it is impractical to consolidate all these databases into a single, uniform format that is optimized for cost, scalability, and performance. Current multi-modal databases attempt to support multiple data formats but often compromise on one or more aspects of cost, performance, and scalability.
For transactional applications to effectively leverage insights, it is crucial to ensure that information from various analytics systems is available in a format that is easily consumable by these applications. The underlying analytics systems must expose insights in a scalable manner that does not introduce unnecessary latency, thereby preserving the customer experience. Additionally, it is imperative for these analytics systems to ingest transactional data in real-time or near real-time, rather than relying on traditional batch processing methods.
So the question is how to create data powered applications (or have data driven application development). One architecture pattern that can be extended to achieve the same is DataMesh. Data Mesh with its principles around Domain Driven Ownership, Data as a Product, Self-Serve Data Infrastructure Platform and Federated Computational Governance provides solid foundation for Data Platform and can be extended to support Data Driven Application Development. One can know more about Data Mesh here
Data Mesh proposes to apply product thinking on Data to offer Data Assets as a Product (Data Product in short). Data Product has a clear business value proposition, aligns to a Domain and defines its own SLOs and adheres to those. Data Products comply with FAIR principles of findable, addressable, interoperable and reusable along with being secure, valuable and trustworthy. Data is the core for Data products and are typically being consumed by business intelligence and analytics applications often in self-service mode. Each data product provides multiple rows and columns of data which can be further used for gaining business insights or even to create other Data Products. Given the ‘bulkiness’ of Data Products they cannot be directly consumed by operational applications.
To have insights in the form that is directly consumable by operational applications, we introduce what is called a ‘Micro Data Product’. A ‘Micro Data Product’ while abiding by Data Product principles has additional following characteristics –
1. It offers an API which can be called directly by application. The API response times supports latency requirements of the operations applications.
2. It is answer to a specific question and hence responds with very little to-the-point data. For e.g. there can be a ‘Micro Data Product’ which answers the following question in the context of retailer inventory application — What was the average time taken to replenish product SKU XYZ in last 1 year? Or in the context of movie studio — What was the total ad revenue when a historical blockbuster was aired last year?
In both the above cases, the user can make decisions on when to reorder a particular SKU or to negotiate the rights for a particular blockbuster title with streaming platforms.
A ‘Micro Data Product’ is typically implemented as a micro service API and often fetches data from an underlying Data Product. It is also registered and discovered via a catalogue. Applications call ‘Micro Data Product’ to provide business insights to the users in their day-to-day operation.
These ‘Micro Data Product’ also register as Tools for the Agents to use as part of Agentic AI applications. Overall, the Technical landscape looks like below –
There is also a paradigm shift while designing and developing applications. The designers are no longer constraint by availability of only the transactional information to be displayed. They also avail the vast insights that has been made available by the analytics platform via Micro Data Products.
