This blog was previously published by the author in his personal blog site.

Artificial Intelligence and Machine Learning are becoming wide spread across various industries. It plays a vital role in developing sci-fi sounding stuff such as Self driving cars and more practical things that touch human lives almost every day, like Voice assistants, Chatbots, Smart power grids etc.

In Software testing, transition from traditional approach to CI/CD driven DevTestOps model is essential to cope up with increased release cadence. Test automation has helped keeping up with the required delivery speed, however adapting AI/ML capabilities in test automation will be a go-forward approach.

Evolution of Software testing

To begin with, let’s review how Software testing has evolved over past few years.

As testing evolved, so does the tools & techniques. However to keep up with the pace of change in the technology, testing strategies must reinvent and bring in required change.

Current challenges in Software Testing

From days of slow paced deliveries of waterfall model to the present day era of continuous application delivery; each phase has its own challenges.

• Size of applications & functional workflows makes achieving the complete test coverage always a challenge.
• Frequent releases and reduced time for testing.
• Rapid changes in applications resulting in huge maintenance of automated tests, as workflows keep changing.
• Pressure of controlling defect leakage in production

While some of the available tools can be leveraged in addressing these challenges to a certain extent, there is a profound need of an Intelligent Test Automation.

With the sheer amount of data generated by applications, test cases, devices etc,; can we use AI/ML algorithms to make informed decisions? After all AI is all about data!

AI/ML/DL – A brief overview

Artificial Intelligence (AI), Machine Learning (ML) & Deep Learning (DL) are interconnected fields, where latter two helps the former in solving complex problems using a data driven approach.

• Artificial Intelligence (AI) is a method of getting machines to think like humans and help make informed decisions based on the available data.

• Machine Learning (ML) is a subset of AI, which uses set of algorithms to keep feeding data to machines. There are three types of machine learning capabilities:
  • Supervised learning (Labelled data)
  • Unsupervised learning (Unlabelled data)
  • Reinforced learning (Adaptation based on environment)

Software testing will mostly be using “Supervised Learning”, as the data will be of certain type.

• The third subset of AI is Deep Learning, which makes use of neural networks to help machines solve complex problems.

The following diagram depicts workflow for training a machine to make informed decisions using AI/ML and DL techniques.

Smart testing using AI/ML 

In “Current challenges in software testing” section, we noted three most commonly seen problems by Software Quality professionals. Let’s explore a possibility to build smart, intelligent tools/frameworks that use data (ML) and help bring intelligence (AI) in Software testing to solve these problems.

Problem Statement #1: Ensuring test coverage, in fast paced product delivery cycle: Modern applications  are required to support multiple platforms such as Deployment configurations, Mobile devices, Web browsers, Operating systems, Multi cloud environments etc., essentially making their compatibility matrix quite large. This directly impacts the number of workflows the applications need to support in order to cater to variety of different use cases. With reduced development and release cycles, it becomes essential that QE teams must ensure testing is complete within stipulated time with the required coverage achieved. Automation is largely adapted by many organisations, which help reduce manual efforts and in-turn the delivery times. However ensuring the coverage still remains a challenge.

How Smart testing tools can help?

Build process, Unit tests, Code coverage all generate variety of artefacts. This information can be used to ensure we are testing enough, in following ways:

• • Group of expert manual testers, will do exploratory testing of the application. Logs (data) generated by this activity can then be fed to ML algorithms to build model for automated tests. This will be an additional step towards ensuring we are covering corner cases in product testing with a touch of human creativity.

Problem Statement#2: Rapidly Changing applications need heavy automation maintenance: Test automation requires maintenance as applications undergo changes over the period of time. In case of agile development, these changes are more frequent during the initial days of product development. To keep up with the development cycle, automation is must to have, however the total time available for testing also gets pulled into the time required to maintain your automated tests.

How Smart testing tools can help?

• AI based, ‘Auto correcting’ automated tests; can keep track of changes happening in the product and make adjustments to test automation code.

Problem Statement#3: Controlling defect leakage in production: Bugs found in production, which ideally should be caught during testing, put QE teams on spot. Therefore serious attempts are always made to ensure enough testing is done. However considering the differences in test V/s production environments, application workloads etc. not every single workflow can be tested.

How Smart testing tools can help?

• Many applications upload logs as part of their call-home capability. While these logs hide the sensitive information, they likely to give good insight to understand application usage patterns. By doing analysis of these logs, Smart testing frameworks can figure out application usage patterns and use this information to fine tune functional/non-functional test suites.
• Smart regression suites can be created by analysing code changes while fixing bugs or enhancing functionality. The insight gained can be used to focus more on weak areas in the product.

Choosing the right tools

Although the concept of using AI/ML in software testing is relatively new, there are handful of commercial tools available in this space. Most are catering to UI Testing needs.

Frameworks, libraries available in opensource can be used to build machine learning models. The selection will vary based on the technology stack, availability of programming skills etc. However the general guideline is to choose a Programming language which is flexible and has built-in support for extensive libraries and frameworks.

Choosing the right tools is just one part in enabling AI/ML in software testing, equally critical is learning required skills. Application of AI in any domain requires following skills and software testing is not an exception.

• Programming – Python & R are most commonly used.
• Data science – Big data concepts, Data modelling etc.
• Mathematics & Statistics

Conclusion 

Though AI/ML is around for a decade. However wide spread use of Internet connected smart devices, Wearable tech etc. is making it a commodity now. Two most important driving factors behind acceptance of AI/ML is; “Data” & “Tools” that can leverage it to make meaningful insights. In current times, both are available in abundance. Software testing rely heavily on Data to make important decisions such as product release – go/no-go and with the right strategy, can leverage AI/ML based tools to make these decisions with greater confidence.