Configuring Elasticsearch

Elasticsearch is a system that allows storing data of various formats and performing full-text search for the data. This article contains a description of a way to configure Elasticsearch for SAYMON.

The suggested manual uses Logstash for gathering and filtering data and Kibana as a graphical interface for Elasticsearch. The three mentioned tools form the so-called ELK (Elasticsearch, Logstash, Kibana) stack.

All of the instructions mentioned below have to be applied on the SAYMON installation server.

Configuring and running ELK

If Docker Compose has not been installed on your system, it is necessary to install it, following the instruction.

In order to run ELK the following steps must be performed:

Download and unzip the elk.tar.gz archive.
Go to the directory elk and execute the following command:
```
$ sudo chmod 777 elasticsearch/data/
```
Run ELK:
```
$ sudo docker-compose up -d
```

After the running, Logstash automatically creates an index named my_index for storing data. Then the index needs to be configured.

To configure the index it is necessary to open the console in Kibana (https://YOUR_HOST_IP:5601/app/kibana#/dev_tools/console?_g=()) and run the following command:

PUT _template/my_index
{
    "index_patterns": [
        "my_index*"
    ],
    "settings": {
        "index": {
            "lifecycle": {
                "name": "my_index-policy",
                "rollover_alias": "my_index"
            }
        }
    },
    "mappings": {
        "dynamic": false,
        "properties": {
            "payload": {
                "dynamic": true,
                "type": "object"
            },
            "entityId": {
                "norms": false,
                "type": "text",
                "fields": {
                    "keyword": {
                        "ignore_above": 256,
                        "type": "keyword"
                    }
                }
            },
            "timestamp": {
                "format": "epoch_millis",
                "type": "date"
            }
        }
    },
    "aliases": {}
}

If the request is successful, the following response will be displayed in the right part of the screen:

{
    "acknowledged" : true
}

Configuring SAYMON to interact with ELK

In order to configure ELK the following steps must be performed:

Add the following parameters to the server configuration file /etc/saymon/saymon-server.conf :
```
"elasticsearch": {
    "node": "http://127.0.0.1:9200",
    "index_pattern": "my_index*"
}
```
Restart SAYMON:
```
$ sudo service saymon-server restart
```

Fetching data from Elasticsearch

Currently, SAYMON users can get data from Elasticsearch only via API (see the Get Stat History method).