Topics In Demand
Notification
New

No notification found.

Making Generative AI Work for Your Enterprise – Excerpts from Recent LinkedIn Live Event
Making Generative AI Work for Your Enterprise – Excerpts from Recent LinkedIn Live Event

September 20, 2023

40

0

Making Generative AI Work for Your Enterprise – Excerpts from Recent LinkedIn Live Event

 

Takeaways from LinkedIn Live roundtable event on ‘Unlocking the Power of Generative AI for Your Organization – A Q&A Roundtable on Large Language Models (LLMs).’

Generative AI and Large Language Models (LLMs) like GPT have the potential to radically transform the way businesses use data and analytics to operate, innovate, serve customers, and grow. LLMs and existing enterprise systems and processes can expedite access to impactful insights and enable personalized content generation for enterprise users and their customers.

The promise of generative AI (Gen AI) is vast. But leveraging the technology in the day-to-day business of the enterprise and driving significant improvements to key business parameters requires a clear definition of what we can expect from the technology and the metrics we plan to use to measure the success of a Gen AI initiative.

I recently hosted LinkedIn Live on ‘Unlocking the Power of Generative AI for Your Organization – A Q&A Round Table on Large Language Models (LLMs)’, an event from the Course5 Compass – GenAI Series, with my co-panelist – Nitesh Jain and Jayachandran K R who provided practical considerations for assessing the actual value of enterprise-level investments in new AI technologies like Gen AI.

Here are some key takeaways from the roundtable event –

How do you measure the success of a Generative AI/LLM deployment?

Defining the right KPIs is the first milestone in measuring the success of a Gen AI implementation.

At a strategic level – Rather than looking at direct ROI, it would be more beneficial to use metrics that measure the impact of the technology on other data & analytics investments and the business. Four key performance indicators (KPIs) you could look at are –

  • Adoption
    Organizations are already generating much data and insights, but not enough people use it. Is the LLM driving higher adoption of data and insights in your organization?
  • Deployment of human resources in more meaningful and impactful activity
    If Gen AI can reduce time to data and insights, can people focus on higher-level tasks that add value to the business?
  • Experience enhancement
    Does it improve the experience of accessing a service for customers and employees and the outcomes of that service?
  • Ability to serve customers faster and better
    Does the LLM enable quicker and more relevant information retrieval and help serve customers better?

At a technical level – If you have to measure the success of a Gen AI initiative, you need to consider whether you are looking for a supercharged performance or an incremental improvement to your existing models/systems already in business-as-usual mode.

If, for instance, you’re looking at replacing your existing chatbot with one powered by LLMs, you would look at a very granular set of metrics as – Is this improving the latency, response time, accuracy, user experience, etc. of my chat service? Is it creating a very engaging and exciting experience for my customers?

Enterprises already have several data and analytics investments. And many Gen AI tools and platforms are coming up every day. What is the right approach for adopting LLMs?

Organizations that have adopted technology without figuring out how it fits into their existing ecosystem have failed to deploy or embrace it. A tremendous amount of investment has gone into the current systems, already driving some business value.

Also, when you think of leveraging LLMs within this system, you can use models from providers like Azure, Google, and AWS; apps, APIs/plugins, and Chrome extensions; or enterprise platforms and open-source finetuned models.

To choose the right approach, you must be clear about what you expect from the technology. Augmentation is the fundamental principle to consider while defining Gen AI/LLM technology expectations.

Augmentation using LLMs can happen at different levels –

  • Basic
    Augmentation can turn dry data and standalone insights into narratives that connect the dots in the business.
  • Intermediate
    Augmentation can drive interaction and engagement with the data, so a user can drill down and ask for more refined information to make/test a hypothesis, find causal connections, derive predictive insights based on actual/simulated data, etc.
  • Advanced
    Augmentation can provide business and R&D teams with new ideas, product suggestions, and questions they haven’t considered! Here the AI program acts as a research assistant, suggesting possible scenarios and things the user should care about during strategic planning.

Using Baselines & Benchmarks

Your current business performance can act as a baseline to judge performance improvement or augmentation, and every model will have its performance benchmarks. Consider which approach will bring you substantial incremental gains on your current baseline and has the scope to scale. Then pick the right path for you.

What you can get from an out-of-box model from Azure, GCP, or AWS versus an enterprise model will be very different, for instance. You can start with the open source models, then assess where you need prompt engineering, finetuned models, larger systems within which Gen AI is integrated, and so on.

Can you give us some use-case examples?

Generative AI is such a powerful technology that teams across the enterprise may be looking to leverage it in some way. They could search massive data sets in seconds, intelligent summarization, content generation, emotional analysis, sentiment analysis, topic modeling, etc.

Data Science teams can use Gen AI copilots.

R&D and Insights teams sitting on a wealth of structured and unstructured information can use it to provide topics and supporting data required for any specific R&D initiative.

Marketing and E-commerce teams can use Gen AI to generate personalized, customized product content and messaging. By combining brand messaging guidelines with product data, you can automate intelligent custom content generation with inbuilt SEO optimization for better customer traction.

The Pharma industry can leverage Gen AI to cull critical insights from new information on adverse events – in the context of the vast repertoire of existing knowledge on molecules and disease/therapy areas.

The scope of use cases is relatively limitless. To prioritize your use case, you must assess the feasibility, likelihood of adoption, and the extent of impact expected in each situation.

The whole point is to make more ingenious tasks easy for your employees.

What kind of Audit process is required to ensure my benchmarks stay true or improve?

Model audits can be done at various levels – at the core model (model performance validation) level, the model’s application context, and the overall governance of all LLMs in an organization.

Auditing can also happen at the level of the data, that is, what kind of data the model is trained on. This is important because, eventually, a model’s capability is only as good as the data it is trained on.

Most importantly, it would be best if you constantly validated the output of a model with the ground truth. You’ll need to assess the work carefully, give it scores to evaluate its quality and accuracy, and determine if that level or percentage is good enough for the task.

Moreover, these models are constantly evolving. You will need high rigor through processes like model versioning, prompt versioning, etc., to closely monitor and refine their performance over time.

The Larger Picture

Community backing is one of the most critical factors in successfully selecting and deploying any new technology. The community creates excitement about technology, makes iterative improvements, and propels an organization’s innovation agenda forward.

Generative AI becomes the means to a larger goal.

You can watch a recording of the complete event here: Unlocking the Potential of Generative AI for Your Organization.

 

Author

Anees Merchant

Anees Merchant

That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.


At C5i, our teams synergize the insights of our consultants and industry experts with the technical expertise of data scientists, data engineers, business analysts, and visualization specialists. Furthermore, we continually enhance our capabilities through dedicated research and development efforts conducted in our AI labs. Central to our ethos is a client-first approach, underpinned by a culture of relentless innovation and commitment to driving adoption. By prioritizing the needs of our clients and embracing innovation, we ensure that we remain at the forefront of delivering transformative solutions in a rapidly evolving business landscape.

© Copyright nasscom. All Rights Reserved.