The Great Architecture Shift
Enterprise Application & Data Platform Architectures is changing again....
Background
It is clear that Generative AI and Language Models have the potential to be a major game-changer in the years to come. The potential for generative AI and Language models to revolutionize the technological world has become increasingly apparent, with more and more individuals and organizations recognizing the potential it holds.
Some of the common use-cases of large language models and Generative AI which are well understood today are centered around four core capabilities. These include the following:
Conversation (chat) in natural language: Generative AI enables the creation of natural language conversations through the generation of text or speech that mimics human communication.
Smarter search and discovery: Generative AI empowers smart search and discovery by understanding user queries and generating relevant, context-aware responses, enhancing information retrieval processes.
Content generation which includes text, images, videos, audios, code and the translation from one form of content to another.
Automation via agents where the question from the user or other agent from the user triggers an end to end application flow.
When you combine the four core capabilities mentioned earlier with unique data assets and domain expertise, the potential for impact extends to virtually every application, spanning both internet-based and enterprise-level solutions.
It's important to acknowledge that while technologies like large language models (LLMs) and AI-driven architectures hold tremendous potential, they also come with a host of formidable challenges including hallucination, desired vs. actual accuracy, cost, regulatory compliance, data privacy, biases etc. Any application architecture or software which incorporates LLMs as a core functionality therefore need to ensure that additional tools and techniques needed to address the above challenges are dealt with.
2023 has been a hallmark year, where we have begin the journey through this great architecture transition which has impact on all facets of software product building as well as data and analytics platforms/tools. I review the impact on architecture as a whole and more specifically different layers of software/data applications including computer system/hardware architecture, data & data storage, use-case driven impact as well as application architecture. Lets firstly review the business oppurtunities from the eyes of software and data platform/systems vendor.
A word on business oppurtunities as seen by Software Product and Data Platform/Tools Vendors
Not to be left behind on making the best of the new opportunity, software product vendors, data platform and data tools vendors are also reviewing the impact of this technology on the specific use case that they are addressing.
There are 2 immediate business opportunities that are driving the existing vendors to closely look at generative AI closely.
Firstly, use of generative AI to power smart AI features that can provide an edge in the market segment. For example…
Help users discover and engage with the tool through a natural language or conversational interface. Common examples of features here could be Smarter AI search or a co-pilot which essentially enables QnA with the user data/functionalities of the underlying product.
Use the generative capability of LLM (text-to-sql or data summarization are some of the use cases here)
Furthermore, a data platform or tool vendor is likely to explore ways in which they can empower customers to create a wider range of AI use cases or data products based on Large Language Models (LLMs), leveraging the capabilities of foundational models. These capabilities can be broadly categorized into three different variants:
Building data applications on existing commercial endpoints. A typical RAG-based application follows a workflow that introduces at least two new components:
A vector database with an embedding server (either internal or API-based).
An LLM API proxy or server (if self-hosting is preferred).
Developing agent-based AI applications that involve function calls to automate specific end-to-end processes.
Fine-tuning domain-specific models on top of the base foundational models (open or closed).
Impact on Data & Application Architecture
Having looked at the business drivers and opportunities, architectural shift is now a foregone conclusion.
To be AI-powered/AI-enabled, at minimum, each of the applications now need to introduce new capabilities such as vector database and language models in their application service architecture and also introduce capabilities such as co-pilot or similar user interfaces for user to exploit those capabilities.
At the other extreme, a large enterprise may have 100s of potential use cases where it can exploit the capabilities of language models. These use cases will have many fundamentally newer requirements on the data management and analytics platform.
Lets look at specific use-cases from “Data and AI platform” point of view.
Use cases for Data and AI Platform
Data and AI Platforms broadly server the following use-cases. Generative AI has an impact on each of these layers as well as the potentially newer use-cases.
Data Cleanup & Enrichment: Often used by creating Master tables/datasets which are then used by downstream use cases such as analytics or reporting or ml model use-cases.
Dashboards & Reporting: Dashboards and Reporting helps businesses stay on top of their business metrics often using visualization. These dashboards typically help answer questions such as “what is happening” or “what has happened” enabling users to carry out business performance analysis. A standard technique employed here is OLAP or metrics models.
Data Analytics: Often questions which are still “what is happening” or “what has happened” or in very few cases may include “light weight forecasting”. These use cases often need much more deeper modeling expertise than what one can do with standard OLAP techniques.
Machine Learning Models: These use cases primarily fall in the realm of classification, regression, forecasting or optimization and tend to use classical Machine learning techniques such as supervised or unsupervised learning or other statistical forecasting or optimization techniques. The practical applications in enterprises today are still in the realm of building models still with structured data or unstructured data which is largely text. Examples of use cases may include recommendation models, sentiment analysis or a revenue forecasting model.
AI use cases typically use deep learning models and are often employed when an enterprise has other forms of unstructured data such as images, videos, documents (pdf), audio (call records in contact centers) etc.
Changing Share of use-cases
Anecdotal evidence from our experience suggests that today almost 90-95 percent of use-cases in most enterprises fall in the first 3 categories, whereas 5-10 percent fall in the “Machine Learning Models” category and another 0-5 percent fall in the “AI category”.
With Generative AI, there is a high probability the above proportion is changing meaningfully
Firstly, the proportion of AI enabled, AI-powered or AI use-cases rise in terms of their share of use cases.
Secondly, For the first 3 categories(Data Cleanup & Enrichment, Dashboards & Reporting and Data Analytics), there’s likely to be change in user workflows via co-pilot stack. Even though these applications largely work with tabular data at the core, they will now have to also keep track of vector embeddings, logging the conversation between user and AI bot for analysis and reducing hullicination. In effect these use-cases now need to deal with newer kinds of data formats as well as database requirements.
Thirdly, as the proportion of last 2 categories(ML Models and AI models) grow, it’s likely that share of non-tabular data such as images, audio, video, pdf or xml file. This would therefore need data platforms to cater to storing and governance layers to support multiple modalities and formats, as they are used by downstream workflows for model training, test and validation also for storing logs for operational data and llm conversations for analysis.
Modal Architecture (Multimodal) use cases
Multi-modal models are advanced AI models designed to understand and generate content that combines multiple modalities, such as text, images, and sometimes audio or other forms of data. These models extend the capabilities of traditional language models, which primarily work with text, by incorporating the ability to process and generate content that involves various types of media or translate from one form to another. There are a large variety of use-cases including rich text documents (including pdfs) and videos with text and images across a variety of industries and use-cases where understanding modalities can be crucial to contextualize and better understand the provided information. For example, understanding medical imaging data along with other unstructured documents/notes can be crucial in generating a concise transcript of a patient's case file.
Building effective multi-modal models and deploying them is challenging and still an evolving field.
But with time Data and AI platforms may need to prepare for dealing with a wider range of data types and ensuring that the downstream applications can build and deploy use-cases involving multiple modalities.
AI Agent driven automation use cases
In my mind, It’s likely to be a combination of approaches, which are fit for specific task/use-case and enterprises and application designers will have to choose a model or a set of models that are fit for the specific data product that one is building.
For example, A specific use-case in financial services may need LLM to extract a set of attribute-values from an unstructured property document, bank statements of the user and then send the output to a classical credit risk scoring model api and write the application risk score to the loan officer.
Impact on Data & AI Platform/Applications
Lets look at specific use-cases from data and AI platform point of view. Impact of the above will be felt at all the layers of platform architecture, be it hardware/compute, storage/data layer, processing layer or application layer.
Hardware (CPU, GPU & Compute Systems)
This is largely applicable to enterprises which have use-cases demanding one of the following scenarios.
AI Inference workload: Companies may want to host their own inference server based on open source LLMs for privacy, control and TCO reasons.
Fine-tuning Impact: More companies take to building/fine-tuning proprietary task specific models.
Research Impact: We likely see research in new emerging model categories such as reasoning models, multimodal AI models and potentially newer model architectures accelerate.
Cloud based orchestrators: Data Science and AI Platform vendors are likely to include cloud based orchestrators for GPU based workloads right within their platform architecture to enable their customers to integrate these powerful machines right within their inference workflow or fine-tuning workflow.
Data & Data Storage Impact
Tabular vs Non-Tabular Data
Data platforms historically have been centered around structured data, and data warehouses have played a crucial role in enterprise data management. Machine learning (ML) and artificial intelligence (AI) use cases that require unstructured or semi-structured data have often been distinct from traditional data warehousing platforms.
As ML/AI use cases continue to evolve and become more diverse, there is indeed a growing need for a more flexible and unified approach to data storage and processing. The concept of a "lake house" architecture pattern has gained significant attention and is seen by many as a promising solution to address these evolving needs.
The lake house architecture pattern represents a holistic approach to data and AI platform design, aiming to bridge the gap between traditional data warehousing and modern data lake capabilities. This unified approach is well-suited to meet the evolving demands of multi-modal and combinatorial ML/AI use cases by providing a versatile, cost-effective, and scalable foundation for data storage, processing, and analytics. It allows organizations to leverage the full spectrum of their data for insights and innovation while maintaining control and governance over their data assets.
Processing Systems (GPU vs CPU)
There is also an inquisitive case for investigating the role of GPUS in processing both sparse and dense dataframes and the potential impact on processing paradigms and distributed processing engines.GPUs today excel in processing sparse and dense dataframe by taking advantage of their efficient and parallel processing paradigm. As GPU availability and awareness grow, there's a compelling case for unifying processing paradigms for sparse and dense dataframes. A unified approach could streamline application development, reduce complexity, and improve performance across various data scenarios.
Such unification might involve designing libraries and frameworks that can seamlessly switch between CPU and GPU processing based on data characteristics. This would enable developers to write code that remains efficient regardless of data density.
Distributed processing engines like Apache Spark and Dask have traditionally been designed to work with CPU-based processing. Integrating GPUs into these frameworks could lead to significant performance improvements for data processing tasks. Another approach could involve hybrid processing, where GPUs are used in conjunction with CPUs to maximize performance.
Application & Services Layer
While LLMs represent a significant technological advancement, they are more likely to be integrated as an important component within existing enterprise application architectures rather than completely displacing the core application framework for a foreseeable future.
A typical software application into its three main components: user interface, application logic, and data layer. Applications must use language models to add intelligence layer each of these components, especially when it comes to powering conversational user interfaces (CUIs) and voice-based interactions. Let's look at likely changes needed for an “intelligence application”
User Interface (UI)
Language models, particularly natural language processing (NLP) models, can be integrated into the UI to create conversational interfaces. These interfaces allow users to interact with the application using natural language.
Chatbots and co-pilots assistants are examples of how language models can be used in the UI. They can answer user queries, assist with tasks, and guide users through complex workflows. Semantic AI Search is another example of how assets can be discovered
Application Logic
Language models can play a crucial role in the application's logic layer by processing and understanding user input. They can interpret user queries, extract intent, and determine appropriate actions.
RAG based systems can be implemented using language models to maintain context and carry out multi-turn conversations within the application.
Data Layer
Language models can assist in data retrieval and analysis. For instance, users can ask questions in natural language to retrieve specific information from the lakehouse.
Text analysis and sentiment analysis models can be used to gain insights from textual data stored in the data layer.
Voice-Based Interfaces
In addition to text-based interactions, voice-based interfaces are becoming increasingly popular. Speech recognition (ASR) and text-to-speech (TTS) models enable applications to accept voice commands and provide voice responses.
This capability is valuable in scenarios like virtual assistants, voice-controlled devices, and applications that require hands-free interaction.
Multimodal Interfaces
Combining language models with computer vision/voice/audio models allows for multimodal interfaces. Users can interact with applications by both speaking and showing images or videos, opening up new possibilities for applications in areas like content analysis and augmented reality.
Improved User Experience
Language models can significantly improve the user experience by making applications more intuitive and user-friendly. Users can communicate with the application in a way that feels more natural and conversational.
Non Functional Requirements
Make no mistake, Data platforms & LLM Applications may need to ensure responsible and effective deployment of LLMs. Let's delve into each of these considerations:
Data Privacy and Regulatory Compliance
Data Access Controls: Implement strict access controls to protect sensitive data used by LLMs. Access should be limited to authorized personnel and applications.
Audit Trails: Maintain comprehensive audit trails to track data usage by LLMs, ensuring accountability and compliance with data privacy regulations.
Observability and Explainability
Request/Response Logging: Implement robust logging mechanisms to capture user interactions and AI bot responses. This data is valuable for analyzing system behavior and diagnosing issues.
Model Interpretability: Data Platform and Applications must incorporate techniques for model interpretability. Understanding why an AI bot generates a particular response is crucial for transparency and debugging.
Reducing Hallucination
Error Analysis: Regularly analyze errors and hallucinations made by LLMs. Identify common patterns and fine-tune models to reduce these errors. Continuous monitoring and feedback loops are essential.
Human Review: Implement mechanisms for human review and intervention when the AI bot's responses are uncertain or could have serious consequences.
Improving Accuracy Over Time
Feedback Loops: Establish feedback loops from users to continually train and refine LLM models. User feedback can be used to identify and correct biases, improve accuracy, and adapt to changing user needs.
Therefore, deploying “Intelligent Applications” over data platforms requires a holistic approach that encompasses data privacy, compliance, observability, error reduction, continuous improvement, and ethical considerations. These measures not only help ensure regulatory adherence but also enhance user trust and the overall effectiveness of AI-powered applications. Continuous monitoring, feedback, and adaptation are key to maintaining the quality and reliability of LLM-based systems.
Conclusion
Advent of generative AI and Large Language Models (LLMs) has ushered in a transformative era for both software applications, data tools and data platform architecture. It is reshaping how we design, develop, and deploy applications, as well as how we manage, govern and leverage data.
It has opened up new horizons for natural language understanding, generation, and automation, which has further led to the integration of conversational interfaces, voice-based interactions, and advanced text analytics within software applications. Users can now engage with applications using natural language, making user experiences more intuitive and inclusive.
Data and Analytics platforms are the backbone of modern enterprise and building data driven applications. They need to evolve to accommodate the diverse data types and processing needs demands by generative AI and LLM driven use cases. This evolution will likely lead to the rise of unified data architectures like the lake house pattern, which seamlessly combine structured and unstructured data and addition of intelligence layer at the core enabling more comprehensive analytics and insights.