Topics In Demand
Notification
New

No notification found.

Blog
Cloud Data Lakes – Four Must-have TCO Optimization Capabilities

July 15, 2020

614

0


Listen to this article



Enterprises leverage cloud providers’ compute and storage services for their ad-hoc data analytics, streaming analytics and ML use cases as cloud data lakes provide significant cost advantages, agility and scale from the get-go. Proof of concepts (POCs) for data-driven initiatives start easily and without any huge upfront bill. But over time as projects mature or ad hoc queries take longer or model iteration cycles increase, the seemingly endless supply of underlying resources leads to wasteful expenditure on compute and resources.

The usage comes with cost unpredictability and lacks financial governance and thus negatively impacts TCO. In the cloud, rising costs are not necessarily bad; it means that the data team is using more services, which theoretically means the team is doing more “good stuff” and hopefully is delivering business value. TCO optimization makes sure that wasteful spending is identified and eventually eliminated. Cloud data lake platforms should be able to help enterprises keep check on this wasteful spending to lower TCO. Admins should be able to do the following for optimizing TCO within their data lake platforms:

  1. Control and design the infrastructure spend at will, override policy, preference, or autonomous self-learning
  2. Leverage built-in capabilities to optimize clusters for lower infrastructure spend based on custom-defined parameters
  3. Monitor total costs at the application, user, account, cluster, cluster-instance level to drive accountability and meaningful discussions across teams
  4. Identify areas of cost optimization to drive maximum performance for the lowest TCO

As platforms provide these core TCO focussed capabilities, it should be autonomous and policy-based TCO optimization without sacrificing service level agreements (SLAs).

With Qubole, the open data lake platform, enterprises address all 4 key above requirements for optimizing TCO by:

  1. Reducing costs continuously in an automated manner based on set or default policy, preference, and autonomous self-learning.
  2. Optimizing the consumption of resources consistently like performance improvements to the underlying engine so that jobs are completed efficiently.
  3. Finding and consuming lower-priced resources on a continual basis with workload-aware autoscaling; admin-defined heterogeneous cluster configurations and only provision resources when needed, whether On-demand or Spot.
  4. Eliminating unnecessary resource consumption with aggressive downscaling, optimized upscaling and at-will shut down.
  5. Throttling queries based on monetary limits based on the budget set by the administrator.
  6. Providing insights for user, job and cluster level cost metrics in a multi-tenant environment to do data-driven show back discussion.

In summary, a cloud data lake platform should be able to understand what is currently happening and build a financial profile of your cloud spending, help put measures in place to control spending and optimize by taking the advantage of cloud data platform facilities to reduce costs and improve overall TCO.

P.S – This blog was first published on https://www.qubole.com/


That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.


QuboleTechnologies

© Copyright nasscom. All Rights Reserved.