Topics In Demand
Notification
New

No notification found.

Blog
Blockchain Powered IoT: Anomaly Detection

July 1, 2018

927

0

Anomaly detection is used widely to perform various tasks such as fraud detection in the financial industry, network breach for cyber-security, and enemy surveillance for the military [1]. Data scientists apply various models to find anomalies, using a range of techniques from statistics to machine learning.  However, the explosion of time series data generated by the Internet of Things (IoT) has made this task more challenging than ever.

Example

The example above shows a data stream which represents machine temperatures recorded over time.  Anomalies are marked by a red circle within the pink shaded anomaly windows [2]. Applying this data to a Machine Learning algorithm will result in automated detection of the anomalies.

It is important to establish some boundaries on the definition of an anomaly [3]. Anomalies can be broadly categorized as:

Point anomalies: A single instance of data is anomalous if it’s too far off from the rest. Business use case: Detecting credit card fraud based on amount spent.

Contextual anomalies: The abnormality is context specific. This type of anomaly is common in time-series data. Business use case: Spending $100 on food every day during the holiday season is normal but may be odd otherwise.

Collective anomalies: A set of data instances collectively helps in detecting anomalies. Business use case: Someone is trying to copy data form a remote machine to a local host unexpectedly, an anomaly that would be flagged as a potential cyber-attack.

Anomaly detection is similar to, but not entirely the same as noise removal.  Noise removal is the process of immunizing analysis from the occurrence of unwanted observations; in other words, removing noise from an otherwise meaningful signal.

The simplest approach to identifying irregularities in data is to flag the data points that deviate from common statistical properties of a distribution, including mean, median, mode, and quantiles. Let’s say the definition of an anomalous data point is one that deviates by a certain standard deviation from the mean. Traversing mean over time-series data isn’t exactly trivial, as it’s not static. You would need a rolling window to compute the average across the data points. Technically, this is called a rolling average or a moving average, and it’s intended to smooth short-term fluctuations and highlight long-term ones. Mathematically, an n-period simple moving average can also be defined as a low pass filter.

The low pass filter allows you to identify anomalies in simple use cases, but there are certain situations where this technique won’t work.  Here are a few:   

1)  The data contains noise which might be similar to abnormal behavior, because the boundary between normal and abnormal behavior is often not precise.

2)  The definition of abnormal or normal may frequently change, as malicious adversaries constantly adapt themselves. Therefore, the threshold based on moving average may not always apply.

3)  The pattern is based on seasonality. This involves more sophisticated methods, such as decomposing the data into multiple trends to identify the change in seasonality.

For the cases 1-3 above, we will need to apply a more sophisticated approach based on Machine Learning.

Machine Learning based approaches will be the subject of a subsequent Blog.

 

For our earlier Blogs and more details about DomaniSystems, visit our Website at https://www.domanisystems.com

 

We encourage open discussion on Smart Contracts, Blockchain, Security and IoT for mutual benefit and enhancement of knowledge and understanding.  You can start your discussion or questions here.  You can also contact us directly; all the information is on our Website.

 

References:

[1]  Detecting Anomalies in IoT with Time Series Analysis, July 26, 2016, Zenrichsocial

[2]  https://numenta.com/resources/papers/unsupervised-real-time-anomaly-detection-for-streaming-data/

[3] Introduction to Anomaly Detection, Pranit Choudhary, 02.14.17

 


That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.


dalemontrone

© Copyright nasscom. All Rights Reserved.