Topics In Demand
Notification
New

No notification found.

Can we search using natural language, just like we would write or speak?
Can we search using natural language, just like we would write or speak?

May 17, 2021

936

0

There is a massive amount of data on the web; how can we effectively search through it for relevant information? And it’s not just the web where we need it: Our computers store terabytes of company and personal data that we need to work with; we need effective search to get our day-to-day job done. And what do I mean by effective search

  • Can we go beyond just matching keywords?
  • Can we search using natural language, just like we would write or speak?
  • Can we make the search smart enough to forgive our minor mistakes?
  • Can we search for things that aren’t an exact match but are “close enough”?

We can answer all those questions with one word: Yes. To understand how, we need to enter the world of Natural Language Processing. NLP is a field of computer science that deals with analyzing natural language data, like the conversations people have every day.

NLP is the foundation of intelligent search, and we have seen three different approaches in this field as follows.

 

Evolution of Search Methods

  1. Rules (1950–1990s)
    Complex handwritten rules that emulate Natural Language Understanding.
    Drawbacks: Handwritten rules can only be made more accurate by increasing their complexity, which is a much more difficult task that becomes unmanageable over time.
  2. Statistics (1990s — 2010s)
    Probabilistic decisions based on weights, machine learning and feature engineering.
    Creating and managing rules was solved with machine learning, where the system automatically learns rules by analysing large real-world texts.
    Drawbacks: These statistical methods require elaborate feature engineering.
  3. Neural Networks (Present)

Advanced machine learning methods such as deep neural networks and representation learning.

Since 2015, statistical methods have been largely abandoned, and there has been a shift to neural networks in machine learning. Popular techniques using this method make it a more accurate and a scalable alternative. It involves

  • Use of word embeddings to capture semantic properties of words
  • Focus on end-to-end learning of higher-level tasks (e.g., question answering)

 

When you use Neural Networks to make your search smarter, we call this a Neural Search System. And as you will see, it addresses some of the critical shortcomings of other methods.

difer

 

Note that the applications of Neural Search are not just limited to text. It goes well beyond what NLP covers. With neural search, we get additional capabilities to search images, audio, video, etc.

 

While the Neural Search method has become more widespread since 2015, and should be the primary focus area of any new search system, we shouldn’t completely rule out Symbolic (rule-based) Search methods. In fact, using a combination of Neural Search and Symbolic Search may result in optimized results.

 

Applications of Neural Search

Semantic Search -  “addidsa trosers”

misspelled brand and category, still returns relevant results similar to query “adidas trousers”

adidas

 

Search between data types

With Neural Search, you can use one kind of data to search another kind of data, for example using text to search for images, or audio to search for video.

mh

Search with multiple data types

With Neural Search, you can build queries with multiple query data types e.g. search images with text+image

ng

 

Source: Jina AI - https://jina.ai/


That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.


images
Nishant Kumar
Technology Enthusiast

© Copyright nasscom. All Rights Reserved.