Topics In Demand
Notification
New

No notification found.

Data Fabric – mitigating hybrid data landscape challenges
Data Fabric – mitigating hybrid data landscape challenges

September 5, 2022

201

0

       1. Introduction

Imagine running AI/ML workloads while dealing with the enormous data that you have at your disposal. Apart from the complexity, you must also deal with the multitude of data storage locations where your data resides. The well-considered view that emerges is that your data needs as much strong logistics as it requires learning capabilities. If you need data from every nook and corner, there is a need to build a robust data access layer or architecture that can help you with a 3600 view of the customer and enterprise data.

Relating this to ‘data agility’, Gartner defines the Data Fabric as:

“…a design concept that serves as an integrated layer (fabric) of data and connecting processes. A data fabric utilizes continuous analytics over existing, discoverable and inferenced metadata assets to support the design, deployment, and utilization of integrated and reusable data across all environments, including hybrid and multi-cloud platforms.”

A Model Data Fabric Architecture - Gartner

Notice the keywords, integrated and reusable data across multi-cloud platforms. This sums up the aspects of business reality as well as the challenges faced when it comes to management of data. It is borne out of the fact that merely managing data pipelines passively won’t be enough, it is equally important to automate parts of this process by leveraging Artificial Intelligence. The following is by far the most comprehensive representation of Data Fabric. 

While this showcases the need to build a layer of interconnected data, provide a ground for automating data engineering tasks, track alerts and changes to the data pipeline, a Data Fabric is also about reducing human errors, making data management affordable, and streamline data integration techniques.

2. The Data Silo - Omnipresent spoilsport

There is data and then there are data silos. Some of it has been the result of natural progression of data storage mechanisms over the years, some of it has resulted from business necessities. Then, there are multiple applications and cloud environments in use across organizations. Enterprise Data Warehouse (EDW), backed by BI platforms, Hadoop-based Data Lakes, and the like. 

On the other side, there is a range of data consumers – data science, data engineering, ML teams who have varied requirements dealing with supply chain, pricing, logistics & warehouse, personalization problems. But the core requirement across the groups is constant – easy accessibility and visibility to data with querying capabilities. In other words, there is a need to standardize access to data and leverage it for use-cases. This is where a Data Fabric comes in.

With increasing AI/ML workloads, most of which have become mission-critical, it will make lesser business sense to depend on individual applications to do the heavy-lifting. Instead, a Data Fabric promises to be a key support mechanism to run AI/ML workloads successfully and efficiently.

3. How does a modern Data Fabric impact success with Artificial Intelligence?

One of the key challenges in making AI implementations replicable is the inability to industrialize the success of one project/process. Often, it is a case of isolated success which despite becoming a cornerstone, fails to impress when it comes to replicating its impact. Wide-spread embedding of Artificial Intelligence in application/enterprise software and productized AI, courtesy AutoML, there is a need to ensure better access and navigation to stored data, within the enterprise, on the cloud or in data centers. Database is passe and what organizations need is an architecture that can enable high-grade data management and integration.

  • For Data Analytics:
    1. Provides an operational layer which can collect data from various sources and make it available to a variety of users. In other words, a Data Fabric enables data access through one platform and provides a single view of data irrespective of where they are stored.
    2. Democratizes data flows and provides real-time access
    3. A data fabric provides a more powerful layer, rather than just a database that allows for integration, storage, and retrieval of data. It also allows for data processing using advanced analytics, AI/ML tools
  • For AI workloads:
    1. A Data Fabric goes a long way to ensure success with AI workloads. Repeatable success depends on various factors, one of which is data discovery and a top-down view of the stored data

    2. AI requires constant access to accurate data. A Data Fabric allows organizations to decouple data access from disparate data entities and works across silos to ingest the required data
  • Infusing Intelligent Automation to Data Management: 
    1. Enables automated self-service data access, consumption, and ingestion  
    2. Automates data flows and simplifies the creation of data pipelines
    3. Automates orchestration and integration, infuses AI capabilities to the data lifecycle processes
    4. Acts as a key ingredient for Trusted AI, a Data Fabric automatically monitors models and identifies the need to retrain as and when needed. A Data Fabric also leverages governance rules for controlling model lifecycle. 

I am reachable at bandev@nasscom.in

 


That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.


images
Bandev Ghosh
Senior Manager

© Copyright nasscom. All Rights Reserved.