Topics In Demand
Notification
New

No notification found.

Blog
Qubole brings machine learning to the data warehouse.

March 24, 2020

653

0


Listen to this article



Machine Learning Juggernaut

Machine learning continues to be on a tear, fueled by the availability of data and compute capacity – especially in the cloud. A wide range of industry metrics points to the same conclusion. Here are some that I found insightful

  • International Data Corporation (IDC) forecasts that spending on AI and ML will grow from $12B in 2017 to $57.6B by 2021.
  • Machine learning patents grew at a 34% Compound Annual Growth Rate (CAGR) between 2013 and 2017, the third-fastest growing category of all patents granted.
  • Deloitte Global predicts the number of machine learning pilots and implementations will double in 2018 compared to 2017, and double again by 2020.

In our customer base, I see a similar growth trend in machine learning use cases. Of our 200+ customers, almost everyone is already using Qubole for machine learning or is in the process of doing so to identify new revenue opportunities, drive process efficiencies or curtail risks.

Does Machine Learning Need Structured, Semi-Structured Or Unstructured Data?

Whether it’s a human being or a machine, the reality is that structured data is far easier to use. However, with the volume, variety, and velocity of data that gets created, the prevalence of unstructured data or semi-structured data is also a reality. This means that machine learning programs will access both structured and unstructured data and regardless of the vendor rhetoric (and sometimes we are guilty of that too) data lakes and data warehouses will play a pivotal role in machine learning.

Enabling Machine Learning On The Data Warehouse With Qubole

As one of the world’s leading cloud-native platform for machine learning, Qubole enables machine learning using Apache Spark on data stored in any data warehouse.

Earlier this year we jointly built a Qubole-Snowflake Apache Spark integration with Snowflake to enable data teams to build, train and put in production powerful machine learning (ML) and artificial intelligence (AI) models in Spark using information stored in Snowflake. The integration also enables data engineers to use Qubole to read and write data in Snowflake for advanced data preparation such as data wrangling and data augmentation to refine existing Snowflake data sets.

Since its launch, the Qubole-Snowflake integration is used by several of our joint customers to deploy machine learning on Snowflake data leveraging Qubole’s Apache Spark capabilities. We also enable access to popular cloud data warehouses such as Azure SQL Data Warehouse or AWS Redshift.

I want to share a real-life example to demonstrate machine learning on the data warehouse. One of our retail customers is using machine learning to create customer segmentation for their next generation personalization engine. They are using the Qubole-Snowflake connector to read 30 million records, classify these into 1700 different segments based on numerous attributes and write them back into the data warehouse. These granular segments are used for creating a highly personalized product and promotion offers.

The Future

As modern data management practices embrace the co-existence of data lakes and data warehouses, we will continue to see a trend where machine learning platforms evolve to access and process data from multiple disparate platforms for structured and unstructured data to support the digital enterprise.

At Qubole, we continue to pioneer innovation that breaks down the silos that prevent organizations from reaching their data activation potential.


That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.


QuboleTechnologies

© Copyright nasscom. All Rights Reserved.