Topics In Demand
Notification
New

No notification found.

Modernize the Data Ecosystem to Lay the Foundation of an Insights-driven Digital Next Enterprise
Modernize the Data Ecosystem to Lay the Foundation of an Insights-driven Digital Next Enterprise

490

0

Data modernization has become an urgent competitive necessity for businesses to stay ahead of the curve - anticipate market changes earlier, understand customer needs more closely, and take and implement winning decisions faster than the competition.

Signs that you probably need to invest in Data Modernization

  • Deceleration of innovation – unwieldy and clunky data platforms are slowing down innovation; project delivery lead times are increasing.
  • Increasing use of quick-fix tools – solutions that provide immediate benefits but eventually compound problems as they are poorly integrated.
  • Decreasing collaboration and increasing redundancy – lack of connectedness, poor data visibility across functions, and more duplication of effort.
  • Governance breakdown – as the number of disconnected teams working on the platform keeps growing, managing data becomes cumbersome, inefficient, and expensive.


That said, technology leaders need to assess the pros and cons of a modernization exercise. Businesses must study the various avenues for modernization and choose the one that gives them the best cost-benefit balance. As with any change management initiative, it is disruptive and entails focused deployment of resources.

In this article, I will discuss three frameworks/platforms that have helped to leverage data effectively for business success.

The Data Warehouse

The Data Warehouse was probably the first enterprise-level platform to use data for business decision support. It came into its own in the Nineties and at the turn of the new Millennium. As its name implies, it organized data in structured and labeled fields that could be easily accessed and worked excellently. 

The Data Warehouse operates in an Extract-Transform-Load flow to convert information into intelligence. Data is extracted from different sources, transformed into a usable form, and loaded into the warehouse. Users can then query and access insights in different forms, usually charts, graphs, and tables.


Data-driven business intelligence, as a concept, gained massive leverage thanks to the Data Warehouse. However, like its counterpart in the real world, the Data Warehouse’s key drawback is poor scalability. It works on pre-built schema and can take in only structured data. As a result, the data is siloed and not all data is captured.

As the three Vs of data - volume, variety, and velocity - grow, as in today’s age of Big Data, the Data Warehouse becomes unwieldy and inefficient. And data’s fourth V, veracity, suffers in consequence.

This is not to say that the Data Warehouse has outlived its utility. It still works efficiently for businesses that deal with a smaller volume and variety of data and provides excellent decision support intelligence at a relatively lower investment.

The Data Lake

The Data Warehouse’s inherent problems gave rise to the Data Lake, a platform with no hierarchical structure that is more attuned to the needs of Big Data.

A data lake is like a reservoir into which raw data can be poured and stored until needed. It has a flat architecture and takes in data in their native formats - emails, documents, images, audio, video, semi-structured data, such as CSV, logs, and XML, as well as structured data from relational databases.

The extract-transform-load process happens within the Lake itself and data is presented as reports, dashboards, and such, to facilitate better visualization and more accurate analytics, as well as to enable machine learning.

The Data Lake is thus capable of managing the high volume, high Variety, and high Velocity of Big Data.

However, the Data Lake also has its drawbacks. 

enterprise data lake reference architecture

 

Once data is put into the Lake, it becomes monolithic. This limits the knowledge that data analysts can gain from it and increases the risk of valuable information going unnoticed. Its centralized control structure stretches the IT team thin. Projects get delayed, forcing teams to resort to poorly integrated ‘quick-fix’ solutions that eventually compound problems. Consequently, it often ends up as a huge unmanageable data dump yard. Drawing any useful sense out of the Data Lake becomes a complex, expensive, and resource-intensive task.

In response to these problems, the concept of a Data Mesh came into being.

The Data Mesh

Unlike the Data Lake, the Data Mesh is a composite, integrated ecosystem, and not a monolith. It is composed of decentralized subsystems or domains, each managed by a dedicated team. In a sense, you can say that the Data Mesh as a whole is greater than the sum of its parts.

It thus offers several advantages over the Data Lake.

It makes domain experts owners of their data. Thus, there is no danger of valuable nuggets of information being lost or ignored.  

data mesh ecosystem

It treats data as a product and enables a smooth and secure flow of data from producers to users, whether outside or within a Data Lake. In that sense, a Data Mesh may include Data Lakes.

It encourages cross-functional teams and empowers them to operate independently, with little or no support from a central IT function. Collaboration is more efficient, the pace of development accelerates, and projects go live much sooner.

Its decentralized approach allows you the flexibility to choose vendors and technologies that work best for you, without getting locked onto one platform.

A Data Mesh can be deployed for a broad range of needs and for diverse use cases:

•             Migrating applications to the cloud

•             Modernizing data lakes to make data more easily accessible

•             Integrating apps, IoT, and analytics in real-time

•             Streaming data pipelines within or from data lakes

•             Data-in-motion analytics

Data & Analytics transformation journey over the last 15 years:

data analytics transformation journey

Reference: Zhamak Dehghani (Data mesh founder)

About the author:

Author image
Bhagaban Khatai
Data Transformation Leader, ITC Infotech

 

 

 

 

 

 

 

A Technology evangelist with 17+ years of experience as a Global SME for Data and Analytics. Focused on strategic problem solving, change management, and successful execution to achieve the planned results. Professional growth fueled by Strategic Thinking, Solution-Oriented Approach, Trusted Partner, Consulting, and Driving Growth and tangible impact.


That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.


ITC Infotech is a leading global technology services and solutions provider, led by Business and Technology Consulting. ITC Infotech provides business-friendly solutions to help clients succeed and be future-ready, by seamlessly bringing together digital expertise, strong industry specific alliances and the unique ability to leverage deep domain expertise from ITC Group businesses. The company provides technology solutions and services to enterprises across industries such as Banking & Financial Services, Healthcare, Manufacturing, Consumer Goods, Travel and Hospitality, through a combination of traditional and newer business models, as a long-term sustainable partner. ITC Infotech is a wholly owned subsidiary of ITC Ltd. ITC is one of India’s leading private sector companies and a diversified conglomerate with businesses spanning Consumer Goods, Hotels, Paperboards and Packaging, Agri Business and Information Technology. For more information, please visit: http://www.itcinfotech.com/

© Copyright nasscom. All Rights Reserved.