Topics In Demand
Notification
New

No notification found.

Syncing Hyperledger Fabric StateDB (CouchDB) to Elasticsearch
Syncing Hyperledger Fabric StateDB (CouchDB) to Elasticsearch

November 2, 2021

392

0

Hyperledger Fabric is fully decentralized. To store world state it uses a key-value database, like for example, CouchDB. Retrieval of data from CouchDB becomes a bottleneck when complex queries are run on that database. So, ELK (Elasticsearch-Logstash-Kibana) comes for a rescue to get the results faster.

CouchDB enables rich querying against a Fabric ledger’s world state (the cache of current values of entries in the blockchain) when those values are modelled as JSON data.

But when the number of entries in the ledger grows, CouchDB queries take more time to get the results. When a query includes searching or sorting on different fields, CouchDB cannot get results in optimal time.

To overcome this, CouchDB can be synced to Elasticsearch. Hyperledger Fabric uses either Google-created LevelDB or Apache CouchDB® as state databases.

If CouchDB is used, then the blockchain world state data is stored in CouchDB. Below is an example demonstrating how CouchDB data can be synced to Elasticsearch:

Pre-requisites:

1. Hyperledger Fabric setup with CouchDB as state database

2. Create a dockerized setup of the ELK stack. Use the below yaml file to create it.

CouchDB

In the below screenshot we have 3 documents in the world state database, which happens to be called “companydb”.

CouchDB Fauxton console showing the documents under companydb.

Let’s sync these documents in two different indexes in Elasticsearch. To do this, we need to configure Logstash.

Logstash

Logstash reads data from different data sources and sends it to a destination, which in our case is Elasticsearch. It is a data processing pipeline. It has 3 stages: input, output and filter and at each stage one can make use of a plugin to process the data.

To do this, SSH into the Logstash container and follow the steps to install a plugin for receiving and processing changes to a CouchDB database.

Steps to install couchdb_changes plugin:

  • Navigate to /usr/share/logstash/bin
  • Command: ` logstash-plugin install logstash-input-couchdb_changes `
  • Under the directory( /usr/share/logstash/config) open the file and pipelines.yml and update the location of logstash.conf file for the field path.config. In my case I have created the file logstash.conf under the same folder.

pipelines.yml
  • Update the configuration entries in logstash.conf for the input, output and filter plugins according to your need.

logstash.conf

Elasticsearch

The example setup above will replicate the data in “companydb” in Elasticsearch under the two indexes shown (“hrindex” and “softwareenggindex”).

Elasticsearch can store, search and analyze large volumes of data quickly. The main advantages of Elasticsearch are speed, scalability, query fine tuning, data types and support for plugins. This setup is made even more useful to human users by a third technology in the E-L-K stack — Kibana.

Kibana

Kibana is a visualization and analytics platform which lets us visualize data from Elasticsearch. Using this, you can create visualizations in charts, tables and maps.

  1. Login to Kibana console.
  2. Query for indexes. Use the following command to get synced indices:
    GET /_cat/indices/*index

This screenshot below shows the output of this query, as well as one I’ll describe in a moment:

In the above screenshot, the query is run to get the list of indexes in Elasticsearch. On the right side for hrindex the count of documents is 1 and for softwareenggindex the count of documents is 2.
The second query shows the documents under hrindex.

Use GET /_cat/count/<index_name> to get a count of documents in any specific index.

use GET /_cat/indices to get all indices

Hyperledger Fabric with ELK stack is a good combination to get the query results faster.

I hope my insights were helpful to you!

Elasticsearch, Logstash, and Kibana are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. Hyperledger Fabric is a project hosted by The Linux Foundation® and Hyperledger is its registered trademark. Apache CouchDB and CouchDB are trademarks of the Apache Software Foundation.

Author —Sai Raja, DLT Labs


That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.


DLT Labs™ is a global leader in the development and delivery of enterprise blockchain technologies and solutions, as well as a pioneer in the creation and implementation of standards for application development. With a deep track record in innovation and one of the world's largest pools of highly experienced blockchain experts, DLT Labs™ enables the transformation and innovation of complex multi-stakeholder processes.

© Copyright nasscom. All Rights Reserved.