# Introduction

## Welcome

Akumuli is a time-series database with it's own unique storage engine, designed specifically for time-series data. Akumuli database consists of two parts - server application and the storage engine implemented as a library.

## Technical papers

* [Compression algorithms](https://docs.google.com/document/d/1yLsN1j8xxnm_b0oN6rFSgWOnCHP-OlJC5pBKZQwTAPc/pub)
* [Storage engine data structure](https://docs.google.com/document/d/1jFK8E3CZSqR5IPsMGojm2LknkNyUZA7tY51N6IgzW_g/pub)


# API Endpoints

## Statistics

<mark style="color:blue;">`GET`</mark> `http://<host>:8181/api/stats`

This endpoint allows you to get storage statistics of the running Akumuli instance. It's also can be used as a test endpoint to test service availability.<br>

{% tabs %}
{% tab title="200 Storage statistics successfully retrieved." %}

```javascript
{    "volume_0":    {        "free_space": "0",        "file_name": "\/root\/.akumuli\/db_0.vol"    },    "volume_1":    {        "free_space": "0",        "file_name": "\/root\/.akumuli\/db_1.vol"    },    "volume_2":    {        "free_space": "0",        "file_name": "\/root\/.akumuli\/db_2.vol"    },    "volume_3":    {        "free_space": "2027974656",        "file_name": "\/root\/.akumuli\/db_3.vol"    }}
```

{% endtab %}
{% endtabs %}

## Read query

<mark style="color:green;">`POST`</mark> `http://<host>:8181/api/query`

This endpoint allows you to retrieve time-series data from the database. The client should provide valid query. The response will use chunked transfer encoding to return the results. The results are encoded using the RESP protocol.

#### Request Body

| Name       | Type   | Description        |
| ---------- | ------ | ------------------ |
| Query Body | object | JSON encoded query |

{% tabs %}
{% tab title="200 " %}

```
+RESP encoded data
+just like this
```

{% endtab %}

{% tab title="400 " %}

```
-RESP encoded error message
```

{% endtab %}
{% endtabs %}

## Search

<mark style="color:green;">`POST`</mark> `http://<host>:8181/api/search`

This API endpoint can be used to retreive the metadata like series names and tag values.

#### Path Parameters

| Name       | Type   | Description        |
| ---------- | ------ | ------------------ |
| Query Body | string | JSON encoded query |

{% tabs %}
{% tab title="200 " %}

```
+RESP encoded output
```

{% endtab %}

{% tab title="400 " %}

```
-RESP encoded error message
```

{% endtab %}
{% endtabs %}

## Suggest

<mark style="color:green;">`POST`</mark> `http://<host>:8181/api/suggest`

This endpoint can be used to retrieve metric names, tag names, and tag values. It powers autocomplete function of the **akumuli-datasource** for Grafana.&#x20;

#### Path Parameters

| Name       | Type   | Description        |
| ---------- | ------ | ------------------ |
| Query Body | string | JSON encoded query |

{% tabs %}
{% tab title="200 " %}

```
+RESP encoded list or results
```

{% endtab %}

{% tab title="400 " %}

```
-RESP encoded error message
```

{% endtab %}
{% endtabs %}

## List functions

<mark style="color:blue;">`GET`</mark> `http://<host>:8181/api/function-names`

This endpoint can be used to retrieve the list of functions that can be used in queries.

{% tabs %}
{% tab title="200 List of functions" %}

```

absaccumulatecmacusumdiffdivideewmaewma-errorfrequent-itemsheavy-hittersmultiplyratescalesmasma-errorsumtop
```

{% endtab %}
{% endtabs %}


# Getting Started

## Installation

You can install Akumuli using this repository: <https://packagecloud.io/Lazin/Akumuli>. The repository has the packages for the following operating systems:

* Ubuntu 14.04 (amd64)
* Ubuntu 16.04 (amd64 and arm64)
* Ubuntu 18.04 (amd64 and arm64)
* Debian Stretch (amd64)
* Debian Jessie (amd64)
* Red Hat Enterprise Linux 7 (amd64)
* CentOS 7 (amd64  and arm64)

OSX and 32-bit Linux are supported but packages are not provided yet.

Alternatively, you can use this [Docker repository](https://hub.docker.com/r/akumuli/akumuli/).

### Building from source

#### Ubuntu / Debian

**Prerequisites**

**Automatic**

* Run `prerequisites.sh`. It will try to do the best thing.

**Manual**

In case automatic script didn't work:

* Boost:

  `sudo apt-get install libboost-all-dev`
* log4cxx:

  `sudo apt-get install log4cxx` or `sudo apt-get install liblog4cxx-dev` on Ubuntu 16.04
* jemalloc:

  `sudo apt-get install libjemalloc-dev`
* microhttpd:

  `sudo apt-get install libmicrohttpd-dev`
* APR:

  `sudo apt-get install libapr1-dev libaprutil1-dev libaprutil1-dbd-sqlite3`
* SQLite:

  `sudo apt-get install libsqlite3-dev`
* Cmake:

  `sudo apt-get install cmake`

**Building**

1. `cmake .`
2. `make -j4`

#### Centos 7 / RHEL7 / Fedora

**Automatic**

* Run `prerequisites.sh`. It will try to do the best thing.

**Manual**

In case automatic script didn't work:

* Boost:

  `sudo yum install boost boost-devel`
* log4cxx:

  `sudo yum install log4cxx log4cxx-devel`
* jemalloc:

  `sudo yum install jemalloc-devel`
* microhttpd:

  `sudo yum install libmicrohttpd-devel`
* APR:

  `sudo yum install apr-devel apr-util-devel apr-util-sqlite`
* SQLite

  `sudo yum install sqlite sqlite-devel`
* Cmake:

  `sudo yum install cmake`

**Building**

1. `cmake .`
2. `make -j4`
3. `make`

## First steps

### Configuration

You should create configuration file first. This can be done using command:

```
> akumulid --init
OK configuration file created at: "/home/username/.akumulid"
```

Now you can edit configuration file `~/.akumulid`. This configuration file contains default settings and comments. Two main configuration parameters are `path` and `nvolumes`. First should contain path to directory when database files should be stored. By default akumuli stores files in `~/.akumuli` directory. You can change this to whatever you like (I'm using `path=/tmp` to run tests most often). Second parameter `nvolumes` should contain number of volumes that akumuli can use to store data. By default every volume is 4GB but this can be changed in configuration.

You can set `--config` flag to provide alternative configuration file location for this and every other command.

### Database creation

Now we can create database itself! Run this command:

```
> akumulid --create
OK database created, path: /home/username/.akumuli
```

You can check that database files is actually created by running `~/.akumuli`. This directory shouldn't be empty. (NOTE: you can delete all this files by running the following command: `akumulid --delete`)

### Configuring Akumuli

Let's return to configuration file (`~/.akumulid`). You can read parameter's descriptions in configuration file. The most important parameters are:

* `path` - tells Akumuli where database volumes should be stored (default value is \~/.akumuli)
* `nvolumes` - number of volumes that should be created (this parameter is only used when you run `akumulid --create` command). If `nvolumes` is set to 0 the storage will be expanded on demand without deleting old data.
* `volume_size` - size of the individual volume (this parameter is only used when you run `akumulid --create` command)
* `HTTP.port` - port used by HTTP server
* `TCP.port` - port used by TCP server
* `TCP.pool_size` - number of threads that should be used to process data (should be less then number of CPUs, if you set this value to 0 the system will try to chose optimal size on start)
* `UDP.port` - port used by UDP server
* `UDP.pool_size` - number of threads that should be used to process data (should be less then the number of CPUs)
* Log4cpp configuration

### Running the server

To run `akumulid` as a server - just run it without parameters:

```
> akumulid
OK UDP  server started, port: 8383
OK TCP  server started, port: 8282
OK HTTP server started, port: 8181
```

Now you can write data through TCP or UDP and read data using HTTP.


# Data Model

## Data model

Akumuli’s data model is designed to track properties of real-world objects. These properties are reffered as **metrics**. They can contain numeric time-series and arbitrary events (represented as strings). Things like temperature and humidity measured over time by the sensors in the room are good examples of such metrics. Another example is a CPU utilization and idle time on the virtual server. In other words, everything that you can draw on a scatter plot can be stored.

![Time-series with associated event](/files/-Lmdwa4-NXJ2i7Svhhh6)

## Tracking objects

Each object is uniquely addressed by the set of tags. For example, the room in the hotel can be pointed by the building and the room numbers (room=42 building=2). So, you can pin down specific room using only two tags. You can select specific objects by specifying both tags in the search query. Or alternatively, you can select all rooms in the specific building by providing only `building` tag.

```javascript
// select specific room
{
    "select": "temperature",
    "where": { "building": 2, "room": 42 }
}
```

The tags can be viewed as an address of the object, but they doesn’t limited to that. You can add redundant tags to increase the search capabilities. In our previous example redundant information like the floor number and the wing of the building (room=42 building=2 floor=1 wing=East) can be added to allow us to select all rooms in some specific part of the building.

```javascript
// select all rooms on the 5th floor of the building #2
{
    "select": "temperature",
    "where": { "building": 2, "floor": 5 }
}
```

Akumuli query system allows you to use only the subset of tags in the query.

## Metric names

Metric is an aspect of the object that can be measured and plotted on the graph. For example, you can measure temperature in the room, voltage in the electric circuit, and amount of free space on disk. Each property has a name usually referred as a **metric (or measurement) name**, e.g. ‘temperature’, ‘voltage’, ‘hddfree’. Metric names should be unique within object. For example, if you want to measure amount of carbon monoxide in the room using two sensors you should use two different measurement names for both sensors. Otherwise you won’t be able to distinguish between measurements of both sensors:

```
CO_level_A room=42 building=2 floor=1 wing=East room_type=Luxe
CO_level_B room=42 building=2 floor=1 wing=East room_type=Luxe
```

The other option is to have a tag (but in this case the series will correspond to two different objects):

```
CO_level room=42 building=2 floor=1 wing=East room_type=Luxe sensor=A
CO_level room=42 building=2 floor=1 wing=East room_type=Luxe sensor=B
```

### Event names

Events names are associated with metric names by adding '!' prefix. For instance, if you have an object with `CO_level` metric, you can also have an event stream associated with it using `!CO_level` name.

## Series names

The combination of metric name and tags defines unique **series name**. Akumuli expects metric name to be followed by the set of tags. The particular ordering of tags doesn’t matter. This is an example of the series names:

```
temperature room=42 building=2 floor=1 wing=East room_type=Luxe
humidity    room=42 building=2 floor=1 wing=East room_type=Luxe
colevel     room=42 building=2 floor=1 wing=East room_type=Luxe
```

These are three series names. All correspond to the same object (room) but different properties. The first one tracks temperature in the room, the second one tracks humidity, and the last one is a carbon monoxide level in the room.

Another example of the series name from the DevOps monitoring world:

```
mem.commit OS=Ubuntu_16.04 region=ap-southeast-1 host=PG-mirror host_IP=172.16.254.1 team=NJ instance-type=m3.2xlarge arch=x64 rack=64
```

This one tracks memory usage on the server. Note that the set of tags is largely redundant. It’s enough to have only “host” or “host\_IP” tag to uniquely identify the series. All other tags can be used to provide rich analytics, e.g. you may want to know the average memory use by instance type. Note that two series names correspond to the same object only if their tags are equal. This also mean that if you add (or remove) one tag to series name, the series name will be different. But you can group some specific tags together in query. For instance, if you have several carbon monoxide sensors in the room:

```
CO_level room=42 building=2 floor=1 wing=East sensor=A
CO_level room=42 building=2 floor=1 wing=East sensor=B
```

You will be able to collapse these two series into one that contains readings from both sensors using the query language:

```
CO_level room=42 building=2 floor=1 wing=East
```

## Data points

Each data point should contain full series name, the timestamp, and the value. The series name, as described above, should contain metric name (or event name) and the set of tags.

```
<metric-name> <tag1>=<tag-value1> <tag2>=<tag-value2>...<tagN>=<tag-valueN>
```

The particular order of the tags doesn’t matter, both “metric tag1=1 tag2=2” and “metric tag2=2 tag1=1” correspond to the same series. The metric name, tag names, and tag values can contain any characters except the space character. The series name `mem commit OS=Ubuntu 16.04 region=ap-southeast-1 host=PG-mirror host_IP=172.16.254.1` is invalid because metric name (mem commit) and one of the tag values (Ubuntu 16.04) contains space character. Also, there shouldn’t be any spaces inside the key-value pair before or after the ‘=’ symbol. The tag “region = ap-southeast-1” is wrong, it should be encoded as “region=ap-southeast-1”. The spaces in series names can be escaped using the '\\' character. For instance: `mem\ commit OS=Ubuntu 16.04 region=ap-southeast-1 host=PG-mirror host_IP=172.16.254.1` is a legit series name because space in metric is escaped.

The series shouldn’t be created beforehand. If you write a data point with new series name Akumuli will create new series automatically.

The timestamp can be a simple integer or datetime in ISO 8601 format (Akumuli only supports combined datetime representation in basic form, like in 20170405T123000.000001001).

All data points of each series should be sorted by the timestamp. You can write to the different series in any order but each series should receive datapoints with increasing timestamps. If you’ve written a data point with the timestamp set to 20170405T123000.099 to some series and now you’re writing a new data point with the timestamp field set to 20170405T122959.001 you will get a “late write” error.

The **metric value** can be an integer or a floating point number. The recommended formatting method is to use a “%.17g” format string (using printf syntax) or any equivalent. This guarantees that the precision won’t be lost. Subsequent metric values can have equal timestamps. Nanosecond precision is supported otherwise.

The **event value** can be a string up to 1KB in size. Subsequent events in the same series should have increasing timestamps with microsecond precision.


# Writing data

Akumuli protocol is based on [redis protocol](http://redis.io/topics/protocol) (aka RESP or REdis Serialization Protocol). The protocol is designed to be simple to implement, human readable, and fast to parse. Apart from RESP, Akumuli supports [OpenTSDB telnet-style API](http://opentsdb.net/docs/build/html/api_telnet/put.html).

## Serialization

Akumuli borrows RESP serialization format but not the protocol. Serialization format uses five data types:

* String
* Integer
* Error message
* Array
* Bulk string

### String

Simple string starts with **+** character followed by characters of the string and terminated by CRLF (carriage return and new line characters), or just by new line. The length of the string is limited by 1 kilobyte. The string can't contain a new-line or carriage return characters.

```
+proc.net.bytes\r\n
```

The strings are used to represent time stamps, floating-point values, and series names.

### Integer

Integer value starts with colon character, followed by the series of digits, and terminated by the CRLF sequence.

```
:314159\r\n
```

Integer values in Akumuli are limited by 84 digits. They are used to represent values and timestamps.

### Error

Error starts with minus character, followed by the series of characters that contains error message, terminated by CRLF.

```
-RESP Error:  declared object size is too large
```

This data type is used to return information about errors to the client.

### Array

Array is a compound data structure that can be composed from arbitrary number of values of different types.

* Array starts with asterisk, followed by the number of elements in the array, terminated by the CRLF sequence.
* A RESP encoded value for every element.

```
*2\r\n
:1\r\n
+second\r\n
```

### Bulk string

Bulk strings are used to represent arbitrary large (up to 1MB) binary objects.&#x20;

* The value starts with **$** character, followed by the length of the string, terminated by the CRLF sequence (or a single new line character).
* The actual bulk string data, terminated by the CRLF. The length of the string should match the value after **$** character.

```
$11\r\n
Hello world\r\n
```

## Writing measurements by one

This is the simpliest possible way to write data to Akumuli instance. You should send each data point individually. Each data-point contains the following:

* Series name
* Timestamp
* Value

### Series name

Series name have the following format: `<metric-name> <tags>`, where `<tags>` is a list of key-value pairs separated by space. You should specify both metric name and tags (at least one key-value pair), otherwise it's not a valid series name. It should be encoded using the [RESP string](/writing-data#string).

Examples:

```
+cpu_user host=hostname region=NW\r\n
```

Here, "cpu\_user" is a metric name and "host" and "region" are tags.

```
+!network.tcp.packet_loss host=postgres\r\n
```

Event name is "!network.tcp.packet\_loss" and tag "host" is set to "postgres". You can find more information about series names in the [Data Model](/data-model) section.

### Timestamp

Timestamp should be in UTC time (Akumuli can't work with local time). To encode the timestamp you can use RESP [string ](/writing-data#string)or [integer](/writing-data#integer).

If the **string** is used, Akumuli will try to interpret its value as ISO 8601 encoded UTC date-time (with nanosecond precision or lower).

```
+20141210T074343.999999999\r\n
```

Note: only basic ISO8601 format is supported. All timestamps are expected to be UTC timestamps. Timezones are not supported.

If the **integer** was used the value will be interpreted as a 64-bit timestamp with nanosecond precision. The beginning of the epoch is Jan-1 1970 (Unix epoch).

```
:1418224205000000000\r\n
```

Note: event timestamps are stored with microsecon precision.

### Metric Value

Value can be encoded using the RESP [string ](/writing-data#string)or [integer](/writing-data#integer). If the **string** is used, it will be interpreted as a string representation of the floating point value. Scientific format is supported.

```
+3.14159\r\n
```

Note that the string representation of the floating point value may loose precision. You may use something like this to print floating point numbers without precision loss:

```
printf("%.17g", float64);
```

If the **integer** is used, it will be interpreted as is. Note that Akumuli uses double precision floating point numbers, defined by IEEE 754 standard. Thus, only 54-bit integers can be represented precisely.

### Event value

Event value can be encoded using RESP [string](/writing-data#string).&#x20;

```
+1m ipv4 listen drops
```

Note that the value will be interpreted as an event value only if the message contains event name that starts with '!' symbol.

### Composing the message

Each individual message should be started with series name, followed by the timestamp and the value.

Full message can look like this (\r\n is replaced with real newlines):

* Metric with string timestamp, integer value and string id with two keys:

  ```
  +balancers.memusage host=machine1 region=NW
  +20141210T074343.999999999
  :31
  ```
* Metric with integer timestamp, string value and string id with one key:

  ```
  +balancers.cpuload host=machine1 region=NW
  :1418224205000000000
  +22.0
  ```
* Event with string timestamp, string value and string id with two keys:

  ```
  +!net.tcp host=machine1 region=NW
  +20141210T074343.999999999
  +1m ipv4 listen drops
  ```

**Important:** even the last line in the stream must be terminated with CRLF.

You can concatenate multiple messages together and send them via TCP connection. Akumuli reads the incoming stream of data, parses it and writes individual data points.

## Error messages

Akumuli doesn't send anything back in response to your writes if everything is OK. But if anything goes wrong it will send back an error message. Client can read data from socket (but not obliged) asynchronously to receive error messages. Error messages are RESP-encoded and will start with '-'.

## Workflow

To send data to Akumuli the client should open a TCP connection and start sending the data. Optionally, client may start to read data from socket in parallel. Akumui will send anything back only in case of error. I case of error, Akumuli will send error message and close the connection.&#x20;

### Late write

At the moment, Akumuli can only write ordered data. The data-points from each individual time-series should be ordered by timestamp (it's OK to write data-points from different series out of order if individual series are ordered by time-stamp).

## Writing measurements in bulk

Bulk transfer can reduce the transfer size. It allows to minimize redundancy in transferred data by grouping together values with the same timestamp and origin.&#x20;

```
+mem.usage host=machine1 region=NW\r\n
+20180102T000200\r\n
+87.4\r\n
+cpu.user host=machine1 region=NW\r\n
+20180102T000200\r\n
+22.1\r\n
```

In this example, *"mem.usage host=machine1 region=NW"* and *"cpu.user host=machine1 region=NW"*  originate from the same host. They share the set of tags and timestamp (because the collector generated both measurements at the same time, with hundred of other measurements like this). The only difference between two measurements are the metric name (cpu.user vs mem.usage) and value. In situation like this, it's logical to send the values in bulk, without duplicating the tags and timestamp for each value (especially, when we have dozens of values, instead of two). Using the bulk format this example will look like this:

```
+mem.usage|cpu.user host=machine1 region=NW\r\n
+20180102T000200\r\n
*2\r\n
+87.4\r\n
+22.1\r\n
```

To write data to Akumuli in bulk format you should specify a compound series name, timestamp (integer or string), and [an array](/writing-data#array) of values (each value can be integer or string).

### Compound series name

Compound series name format contains pipe delimited list of metric and event names followed by the set of tags:

```
<name1>|<name2>|...|<name3> <tags>
```

List of metric/event names inside the compound series name shouldn't have any spaces. On the storage side this will be converted to the list of series names: `<name1> <tags>`, `<name2> <tags>`, etc.

```
+cpu.real|!cpu.real|cpu.user|cpu.sys|!net.tcp host=machine1 region=NW
```

This encoding is more compact and saves a lot of network bandwidth.

### Timestamp

Series name should be followed by the timestamp, the same way as [described previously](/writing-data#timestamp).

### Array of values

The list of values is represented using the [RESP array](/writing-data#array). RESP array starts with **\*** symbol followed by the number of elements in the array. This number should match the number of metrics in the compound series name! This number should be followed by values (number of values should match the number of metric names and all values should have the same order, e.g. if cpu.real goes first in the compound series name it's numeric value should go first in the array).

Example:

```
+cpu.real|cpu.user|cpu.sys host=machine1 region=NW
+20141210T074343
*3
+3.12
+8.11
+12.6
```

This will produce three writes:

* Series name: cpu.real host=machine1 region=NW, TS: 20141210T074343, Value: 3.12
* Series name: cpu.user host=machine1 region=NW, TS: 20141210T074343, Value: 8.11
* Series name: cpu.sys host=machine1 region=NW, TS: 20141210T074343, Value: 12.6

## Dictionary mode

In many situations client knows what series it will be sending in advance. This can help to cut down the redundancy in transferred data by providing the series name dictionary. Consider this example:

```
+mem.usage host=machine1 region=NW\r\n
+20180102T000200\r\n
+87.4\r\n
+mem.usage host=machine1 region=NW\r\n
+20180102T000201\r\n
+87.5\r\n
+mem.usage host=machine1 region=NW\r\n
+20180102T000202\r\n
+88.1\r\n
```

Here, the lines that change are the timestamps and values. The series name is always the same. The client needs to send it every time, and Akumuli have to parse it for every data-point. To minimize this expenses, client can send the pre-computed dictionary for series names. This dictionary maps series names to user-defined integer ids. This ids then can be used instead of series names in protocol (this also works with bulk protocol).

### Sending the dictionary

The dictionary can only be sent in the beginning of the TCP session. The dictionary is represented using the [RESP array](/writing-data#array). Every key-value pair is represented using two consecutive elements of the array. The series name is represented using the [RESP string](/writing-data#string). Series name should be followed by the unique id. Id should be represented using the [RESP integer](/writing-data#integer).&#x20;

```
*4\r\n
+cpu.user host=machine1 region=NW\r\n
:1\r\n
+mem.usage host=machine1 region=NW\r\n
:2\r\n
```

This dictionary can later be used in the transmission. To use the series name from the dictionary one should use it's id, in place of series name (or compound series name). The id should be encoded using [RESP integer](/writing-data#integer).

```
:1\r\n
+20180102T000200\r\n
+11.2\r\n
:2\r\n
+20180102T000200\r\n
+43.99\r\n
```

Here, the first data point corresponds to **cpu.user** series, and the second one to **mem.usage**.

You can send the dictionary using a single array or several consecutive ones. The only requirement is that the ids should be unique for the TCP-session, and that the dictionary should be sent first. The following data-points can use or ignore the dictionary.

## OpenTSDB telnet-style API

Akumuli have limited support for OpenTSDB telnet-style API. Only `put` command supported at the moment. The data can be inserted in this form: `put <metric-name> <timestamp> <value> <list-of-tags>`. Example:

```
put cpu.user 1483228800 10.005344383927394 OS=Ubuntu_14.04 arch=x64 host=host_0 instance-type=m3.large rack=86 region=eu-central-1 team=NJ
put cpu.sys 1483228800 9.9992693762580025 OS=Ubuntu_14.04 arch=x64 host=host_0 instance-type=m3.large rack=86 region=eu-central-1 team=NJ
put cpu.real 1483228800 10.002083289000792 OS=Ubuntu_14.04 arch=x64 host=host_0 instance-type=m3.large rack=86 region=eu-central-1 team=NJ
put idle 1483228800 9.9815370970857522 OS=Ubuntu_14.04 arch=x64 host=host_0 instance-type=m3.large rack=86 region=eu-central-1 team=NJ
put mem.commit 1483228800 9 OS=Ubuntu_14.04 arch=x64 host=host_0 instance-type=m3.large rack=86 region=eu-central-1 team=NJ
put mem.virt 1483228800 10 OS=Ubuntu_14.04 arch=x64 host=host_0 instance-type=m3.large rack=86 region=eu-central-1 team=NJ
```

### Timestamp

OpenTSDB uses timestamps with 1-second precision. It's a regular Unix timestamp (number of seconds since epoch). Akumuli can also understand timestamps in this format and all tools that can write data to OpenTSDB will work. But in addition, passing ISO8601-formatted date time or nanosecond precision timestamp will also work.

OpenTSDB endpoint can be disabled in configuration (it's enabled by default in configuration generated by `akumulid --init` command). Note that OpenTSDB endpoint in Akumuli is a bit slower than native TCP endpoint due to protocol verbosity.

### Differences with OpenTSDB

When you're using OpenTSDB you should went to great length in your database schema design. OpenTSDB stores all series with the same metric name together. That's why you should be careful not introducing to many series with the same metric name. With Akumuli this is not needed, since it stores each series individually.

Akumuli implements 'put' command. Other commands, like 'histogram' or 'rollup' are ignored. The 'version' command returns a string that indicates that Akumuli endpoint is used.


# Query Language

Akumuli query language reference.

Akumuli uses REST interface for data retrieval. You should use [**api/query**](/api-endpoints#read-query) endpoint for data queries and [**api/search**](/api-endpoints#search) endpoint for metadata queries.

## Data Model

All data is split between the different metrics. You can think about the metric as a namespace for series names. Each series name starts with metric name:

```
cpu.user host=pg-balancer OS=Trusty arch=x86_amd64 region=NE
```

Here, **cpu.user** is a metric name, and **host=pg-balancer** is a tag-value pair. Akumuli uses [Boolean model (BIR)](https://en.wikipedia.org/wiki/Standard_Boolean_model) for series name searching. For every query you need to specify a metric name and optionally, a set of tag-value pairs (using the [where](/query-language#where-field) field). Series that have this metric name and contains this tag-value pairs will be added to result set. If the query doesn't have a where field, it will return all time-series with the same metric name.

Every series name is linked to the list of data points (the actual time-series data). Every data-point contains a timestamp (64-bit, nanosecond precision) and a value (double precision floating point). You need to specify a search range for that data inside the query.

## Query Object

To retrieve the data you should create a query object that describes what data you need and what shape it should have. The query should be JSON-encoded and sent to ['query' endpoint](/api-endpoints#read-query).

## Error Handling

Query parsing errors are reported using the [RESP protocol](/writing-data#error-messages). The only line in the response will be started with '-' followed by the error message.

Some errors can be reported using the HTTP error codes (e.g. when the wrong API endpoint is used). The query parsing errors are reported using the error messages and the query processing errors usually reported using the HTTP error codes.

## Query Types

Query object can be of one of the following types:

* [Select query](/query-language#select-query)
* [Aggregate query](/query-language#aggregate-query)
* [Group-aggregate query](/query-language#group-aggregate-query)
* [Join query](/query-language#join-query)

### Select Query

This is a simplest possible query type. It can return raw time-series data without aggregation. Select query can have a post-processing steps (e.g. rate or sliding window computation).

Select query can return more than one series but they should have the same metric name.

| Field                                              | Required | Commentary                                 |
| -------------------------------------------------- | -------- | ------------------------------------------ |
| [select](/query-language#select-field)             | Yes      | Metric name                                |
| [range](/query-language#range-field)               | Yes      | Time range                                 |
| [where](/query-language#where-field)               | No       | Tag filter                                 |
| [group-by](/query-language#group-by-field)         | No       | Merge series by tag (depricated)           |
| [group-by-tag](/query-language#group-by-tag)       | No       | Merge series by tag                        |
| [pivot-by-tag](/query-language#pivot-by-tag-field) | No       | Merge series by tag                        |
| [order-by](/query-language#order-by-field)         | No       | Order of the data-points in the result set |
| [filter](/query-language#filter-field)             | No       | Value based filtering                      |
| [limit](/query-language#limit-and-offset-fields)   | No       | Limit on output size                       |
| [offset](/query-language#limit-and-offset-fields)  | No       | Offset of the query output                 |

### Select Events Query

This query returns all events in specified time-range. This query can be used only with events.

| Field                                                | Required | Commentary                                 |
| ---------------------------------------------------- | -------- | ------------------------------------------ |
| [select-events](/query-language#select-events-field) | Yes      | Event name                                 |
| [range](/query-language#range-field)                 | Yes      | Time range                                 |
| [where](/query-language#where-field)                 | No       | Tag filter                                 |
| [group-by](/query-language#group-by-field)           | No       | Merge series by tag (depricated)           |
| [group-by-tag](/query-language#group-by-tag)         | No       | Merge series by tag                        |
| [pivot-by-tag](/query-language#pivot-by-tag-field)   | No       | Merge series by tag                        |
| [order-by](/query-language#order-by-field)           | No       | Order of the data-points in the result set |
| [limit](/query-language#limit-and-offset-fields)     | No       | Limit on output size                       |
| [offset](/query-language#limit-and-offset-fields)    | No       | Offset of the query output                 |

### Aggregate Query

This query can be used to calculate aggregates over time-series. The query returns only one result for every time-series.

| Field                                              | Required | Commentary                           |
| -------------------------------------------------- | -------- | ------------------------------------ |
| [aggregate](/query-language#aggregate-field)       | Yes      | Metric name and aggregation function |
| [range](/query-language#range-field)               | Yes      | Time range                           |
| [group-by-tag](/query-language#group-by-tag)       | No       | Merge series by tag                  |
| [pivot-by-tag](/query-language#pivot-by-tag-field) | No       | Merge series by tag                  |
| [where](/query-language#where-field)               | No       | Tag filter                           |
| [output](/query-language#output-field)             | No       | Set output format                    |

### Group-aggregate Query

This query is used to downsample time-series data. It divides all data-points into a series of equally sized bins and computes a single value for every bin if the bin is not empty. If the bin is empty it doesn't produce any value. The same aggregation functions that can be used with [aggregate query](/query-language#aggregate-query) can be used with group-aggregate. The difference between the aggregate and group-aggregate queries is that the aggregate produces only one value for every series but the group-aggregate can produce a time-series with fixed step. Also, more than one aggregation function can be used with group-aggregate query.

| Field                                                    | Required | Commentary                                 |
| -------------------------------------------------------- | -------- | ------------------------------------------ |
| [group-aggregate](/query-language#group-aggregate-field) | Yes      | Query specific parameters                  |
| [range](/query-language#range-field)                     | Yes      | Time range                                 |
| [where](/query-language#where-field)                     | No       | Tag filter                                 |
| [group-by-tag](/query-language#group-by-tag)             | No       | Merge series by tag                        |
| [pivot-by-tag](/query-language#pivot-by-tag-field)       | No       | Merge series by tag                        |
| [order-by](/query-language#order-by-field)               | No       | Order of the data-points in the result set |
| [filter](/query-language#filter-field)                   | No       | Value filter                               |
| [limit](/query-language#limit-and-offset-fields)         | No       | Limit on output size                       |
| [offset](/query-language#limit-and-offset-fields)        | No       | Offset of the query output                 |

### Join Query

Join query can be used to align several metrics together. The query will group together series that has the same tags but different metric names. The resulting output will be in [bulk load format](/writing-data#writing-measurements-in-bulk). Series names of the individual series will be joined together using the [compound series name format](/writing-data#compound-series-name).

| Field                                              | Required | Commentary                                    |
| -------------------------------------------------- | -------- | --------------------------------------------- |
| [join](/query-language#join-field)                 | Yes      | List of metrics to join                       |
| [range](/query-language#range-field)               | Yes      | Time range                                    |
| [where](/query-language#where-field)               | No       | Tag filter                                    |
| [group-by-tag](/query-language#group-by-tag)       | No       | Merge sereis by tag                           |
| [pivot-by-tag](/query-language#pivot-by-tag-field) | No       | Merge series by tag                           |
| [order-by](/query-language#order-by-field)         | No       | Ordering of the data-points in the result set |
| [filter](/query-language#filter-field)             | No       | Value filter                                  |
| [limit](/query-language#limit-and-offset-fields)   | No       | Limit on output size                          |
| [offset](/query-language#limit-and-offset-fields)  | No       | Offset of the query output                    |

## Query Fields

The query object is JSON encoded. It can contain the following set of fields. Some of this fields are mandatory and other are optional. The 'select', 'aggregate', 'group-aggregate', and 'join' fields define query type. Query object should have one of them.&#x20;

### Range Field

Both "range.from" and "range.to" fields are mandatory for most queries. They can be omitted in 'aggregate' query. In this case the 'aggregate' query will compute aggregate for the whole time-series.&#x20;

If "range.from" is less than "range.to" the time-series data points will be returned in ascending order (from old to new). If "range.from" is greater than "range.to" then the time-series data points will be returned in descending order (from recent to old).

Range field denotes the time interval that query should fetch.

| Field   | Format                                                 | Description                                                                                   |
| ------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| "range" | { "from": "20180530T123000", "to": "20180530T130000" } | Field should contain a dictionary with two keys, "from" and "to". Both values are timestamps. |

Both timestamps should be encoded using the basic ISO8601 format. The same format is used for data ingestion with RESP protocol. Alternatively, nanosecond precision UNIX timestamps can be used as values.&#x20;

### Select Field

Select field is used to tell Akumuli what metric should be fetched.

| Field    | Format        | Description |
| -------- | ------------- | ----------- |
| "select" | "metric.name" | Metric name |

This field defines the query type. If this field is used the query will be a [select query](/query-language#select-query). You can provide only one "select" field. This field can have only one metric name. The query will fetch all series with this metric name. This series can be further filtered by using "where" field.

### Select Events Field

Select events field is used in 'select-events' query to specify event name.

| Field           | Format        | Descritpion |
| --------------- | ------------- | ----------- |
| "select-events" | "!event.name" | Event name  |

This field defines query type. If this field is used in the query, the query will be a [select-events](/query-language#select-events-query) query.

### Aggregate Field

Aggregate field is required to create an [aggregate query](/query-language#aggregate-query). The type of the field is a dictionary that has the following format:

```
"aggregate": { <metric-name>: <aggregation-function> }
```

At least one metric-name and aggregation-function pair should be provided. The available aggregation functions are these:

| Name           | Description                           |
| -------------- | ------------------------------------- |
| count          | Number of elements in series          |
| max            | Largest element in the series         |
| min            | Smallest element in the series        |
| mean           | Average value                         |
| sum            | Sum of all values in the series       |
| min\_timestamp | The timestamp of the smallest element |
| max\_timestamp | The timestamp of the largest element  |
| first          | The first value in the range          |
| last           | The last value in the range           |

The aggregate query object computes aggregate only for values inside the specified [time-range](/query-language#range-field). If there is no values inside the range, the query will return an error.

### Group-aggregate Field

Group-aggregate field is required to make a [group-aggregate query](/query-language#group-aggregate-query). The field is a dictionary with the following format:

```
{
    "group-aggregate": {
        "metric": <metric-name>,
        "step": <time-duration>,
        "func": <function-name>
}
```

| Field                  | Format | Description                                                     |
| ---------------------- | ------ | --------------------------------------------------------------- |
| group-aggregate.metric | String | Metric name (same as in [select](/query-language#select-field)) |
| group-aggregate.metric | List   | List of metric names                                            |
| group-aggregate.step   | String | Aggregation step (10s, 1h, 5m)                                  |
| group-aggregate.func   | String | Aggregation function                                            |
| group-aggregate.func   | List   | List of aggregation functions                                   |

#### Using one function

#### Using list of metrics

If more than one metric name is used the values with different metric names will be interleaved in the output. All function will be applied to every metric name. For instance if the query has two metrics - 'cpu' and 'mem', and two functions - 'min' and 'max' the output will have the following format:

```
+cpu:min|cpu:max host=host1\r\n
+20170101T221015\r\n
*2\r\n
+0.05\r\n
+99.7\r\n
+mem:min|mem:max host=host1\r\n
+20170101T221015\r\n
*2\r\n
+1073741824\r\n
+8589934592\r\n
```

So the output format is defined only by group-aggregate functions and not by the number of metric names.

If only one aggregation function is used in `group-aggregate` field the output will have the following format:

```
+cpu:min host=host1\r\n
+20170101T221015\r\n
+0.05\r\n
```

The series name of the original series changes. The tags stays the same but the metric name gets the **:\<function-name>** suffix. In the example above, original series name was 'cpu host=host1' but the resulting series name is 'cpu:min host=host1'.&#x20;

#### Using list of functions

If more than one aggregation function was used in group-aggregate field the output will have the following format:

```
+cpu:min|cpu:max host=host1\r\n
+20170101T221015\r\n
*2\r\n
+0.05\r\n
+99.7\r\n
```

Metric name is changed as described above plus, the [compound series name format](/writing-data#compound-series-name) is used. The query will return a series for every aggregation function in the list. This series will have the same timestamps but different values (since different functions were used to produce them). Then, these series will be joined together and the [bulk format](/writing-data#writing-measurements-in-bulk) is used to return them.

### Join Field

Join field is used to make a [join query](/query-language#join-query). This field's type is list. The list should contain valid metric names. Example:

```
{
    "join": ["cpu", "mem", "iops"]
}
```

&#x20;Here `cpu`, `mem`, and `iops` is different metric names. Query processor will find series names with the same set of tags with this metrics and join them. E.g. if we have three series - "cpu host=host1", "mem host=host1", and "iops host=host1" - all three series will be joined together producing single series "cpu|mem|iops host=host1".  The output will contain records in [bulk format](/writing-data#writing-measurements-in-bulk).

```
+cpu|mem|iops host=host1\r\n
+20161231T235500\r\n
*3\r\n
+10.5\r\n
+4870\r\n
+148\r\n
```

This field was replaced with \`pivot-by-tag\` field.

### Where Field

Where field is used to limit number of series returned by the query.

| Field   | Format                                  | Description                                                                                       |
| ------- | --------------------------------------- | ------------------------------------------------------------------------------------------------- |
| "where" | { "tag-name": "tag-value" }             | Include only series names which has tag "tag-name" set to "tag-value".                            |
| "where" | { "tag-name": \[ "value1", "value2" ] } | Include only series names which has tag "tag-name" set to one of the values "value1" or "value2". |

You can specify many tags in one where field. This data in conjunction with metric name (or names) will form be used to [search series](/query-language#data-model) inside the index.

Note that the timestamps and values are the same. Only series names are different.

### Pivot-by-tag Field

In a nutshell, `pivot-by-tag` tells query processor to remove all tags from series name except the ones that was listed. After that all series that have matching tags are considered equal and merged together.&#x20;

For instance, if `pivot-by-tag` field was used to specify a single tag name, all series with this tag with the same value will collapse into one. All data points from that series will be joined together. The resulting time-series will contain all data-points from the original series. The series name will contain only the specified tag. It's also possible to specify more than one tag.&#x20;

### Group-by-tag

The `group-by-tag` tells query processor to remove listed tags from series name. After that all series that have matching tags are considered equal and merged together.&#x20;

| Field        | Format                 | Description                       |
| ------------ | ---------------------- | --------------------------------- |
| group-by-tag | \["tag1", ..., "tagN"] | List of tag to remove from series |
| group-by-tag | "tag-name"             | Tag to remove from series name    |

The `group-by-tag` is the opposite of `pivot-by-tag`. Given the example in previous section query with `"group-by-tag": ["valve_num"]` would produce the following output:

```
+pressure_kPa sensor_num=0
+20160118T171000.000000000
+204.0
+pressure_kPa sensor_num=1
+20160118T171000.000000000
+204.1
+pressure_kPa sensor_num=0
+20160118T171000.000000000
+208.0
+pressure_kPa sensor_num=1
+20160118T171000.000000000
+208.2
...
```

### Group-by Field (DEPRICATED)

| Field        | Format                           | Description                                              |
| ------------ | -------------------------------- | -------------------------------------------------------- |
| pivot-by-tag | \[ "tag1", "tag2", ..., "tagN" ] | The list of tags that resulting series name should have. |
| pivot-by-tag | "tag-name"                       | The single tag the resulting series should have.         |

Suppose that you need to store the valve pressure measurements. Pressure in each valve is measured by two separate sensors so you're end up with this schema: `pressure_kPa valve_num=XXX sensor_num=YYY`. Here we have `pressure_kPa` metric with two tags: `valve_num` and `sensor_num`. If you query this series you will get the following results (*\r\n* omitted):

```
+pressure_kPa valve_num=0 sensor_num=0
+20160118T171000.000000000
+204.0
+pressure_kPa valve_num=0 sensor_num=1
+20160118T171000.000000000
+204.1
+pressure_kPa valve_num=1 sensor_num=0
+20160118T171000.000000000
+208.0
+pressure_kPa valve_num=1 sensor_num=1
+20160118T171000.000000000
+208.2
...
```

Each combination of sensor and valve produces its own time-series. If you want to group data only by valve you can use "pivot-by-tag" field. If you add a `"pivot-by-tag": [ "valve_num" ]` field to the query the result will look like this:

```
+pressure_kPa valve_num=0
+20160118T171000.000000000
+204.0
+pressure_kPa valve_num=0
+20160118T171000.000000000
+204.1
+pressure_kPa valve_num=1
+20160118T171000.000000000
+208.0
+pressure_kPa valve_num=1
+20160118T171000.000000000
+208.2
...
```

### Order-by Field

This field can be used to control the order of the data-points in the query output.

| Field      | Format   | Description                |
| ---------- | -------- | -------------------------- |
| "order-by" | "series" | Sort output by series name |
| "order-by" | "time"   | Sort output by timestamp   |

This field takes single string. It can be "series" or "time". If `order-by` is "series" the results will be ordered by series name first and then by timestamp. If `order-by` is "time" then data points will be ordered by timestamp first and then by series name.

### Output Field

This field can be used to control format of the output.

| Field    | Format                                   | Description                                                |
| -------- | ---------------------------------------- | ---------------------------------------------------------- |
| "output" | { "format": "csv", "timestamp": "raw" }  | Set output format to "csv" and timestamp format to "raw".  |
| "output" | { "format": "resp", "timestamp": "iso" } | Set output format to "resp" and timestamp format to "iso". |

The field is a dictionary with two possible values. The first one is `output.format` . It can be set to "resp" or "csv". The first value is used by default. The output will be formatted using [RESP serialization](/writing-data#serialization) format. The same that is used to send data to Akumuli. The second value changes the output format to CSV. This is how the output of the query will look with `output.format` set to "csv":

```
test tag=Foo, 20160118T173724.646397000, 999996
test tag=Foo, 20160118T173724.647397000, 999997
test tag=Foo, 20160118T173724.648397000, 999998
test tag=Foo, 20160118T173724.649397000, 999999
```

The second field is `output.timestamp`.  It controls formatting of the timestamps in the output of the query. If it's set to "raw" Akumuli will format timestamps as 64-bit integers.

```
test tag=Foo, 1453127844646397000, 999996
test tag=Foo, 1453127844647397000, 999997
test tag=Foo, 1453127844648397000, 999998
test tag=Foo, 1453127844649397000, 999999
```

If it's set to "iso" timestamps will be formatted according to ISO8601 standard.

### Filter Field

Filter field can be used to filter data-points by value.

| Field    | Format                  | Description                                                            |
| -------- | ----------------------- | ---------------------------------------------------------------------- |
| "format" | { "gt": 10, "lt": 100 } | Filter out all values less or equal to 10 and greater or equal to 100. |
| "format" | { "ge": 0, "le": 1 }    | Filter out all negative values and all values greater then one.        |

This field should contain a dictionary with the predicates. The possible predicates are "gt" (greater than), "ge" (greater or equal), "lt" (less than), and "le" (less or equal). It is possible to combine two predicates if you want to read values that fit some range, for instance `"filter: {"gt": 0, "lt": 10 }` will select all values between 0 and 10, but not 0 and 10. You can use only predicate if needed.

The use of filter field can speed up query execution if the number of returned values is small. In this case the query engine won't read all the data from disk but only those pages that have the data the query needs.

#### Multi-dimensional filter

Filter field can be used with the join query. If this is the case, you have to specify the metric to which the filter should be applied.

```
{
    "join": ["cpu", "mem", "iops"],
    "filter": {
        "cpu": { "gt": 200 },
        "mem": { "lt": 100 }
    },
    ...
}
```

In this case filter >200 will be applied to "cpu" metric and the filter <100 will be applied to "mem" metric.

### Limit and Offset Fields

You can use `limit` and `offset` query fields to limit the number of returned tuples and to skip some tuples at the beginning of the query output. This fields works the same as LIMIT and OFFSET clauses in SQL.

Don't use this fields if you need to read all the data in chunks. Akumuli executes queries lazily. To read data in chunks, you can issue a normal query (without limit and offset) and read the first chunk (without disconnecting from the server afterwards). When you done with the first chunk you can read the next one, and so on. The query will be executed as far as you read data through the TCP connection. When you'll stop reading to process the data the query execution on the server will pause. It will resume when you'll continue reading.


# Hardware Requirements

## Memory

Every individual time-series is stored on disk using a memory resident component. This memory resident component is composed using IOVec blocks. Each IOVec structure contains up to 4KB of memory. The number of IOVec blocks needed to handle individual time-series depends on its disk-resident size. Akumuli stores time-series in LSM-tree like structure every level of which allocates a single IOVec block. The number of extents depend on disk space used by time seires. This memory/disk space dependency looks like this:

| Memory | Disk  |
| ------ | ----- |
| 4KB    | 0     |
| 8KB    | 128KB |
| 12KB   | 4MB   |
| 16KB   | 128MB |
| 20KB   | 4GB   |
| 24KB   | 128GB |
| 28KB   | 4TB   |
| 32KB   | 128TB |
| 36KB   | 4PB   |
| 40KB   | 128PB |

The individual data element can occupy from 0.1 to 9 bytes, depending on randomness of the data. For many monitoring workloads that number can be in 1-2 byte range (integer values). This mean that time-series with hundreds of millions values will eat up less than 16KB of RAM.

IOVec blocks are allocated with 1KB step. So for the 1-element time-series only 1KB will be allocated. When series won't fit into 1KB the second 1KB chunk will be allocated and so on, until IOVec block won't be filled. This means that the numbers in the table above are worst case numbers. On average IOVec blocks are half full on every level. For instance, this means that for the series that occupies from 128MB to 4GB on disk Akumuli may need to use 5-20KB for the memory-resident component. Averaged for high number of individual time-series that will give us 11KB of RAM per/series in this specific scenario.

The expected memory requirements **per-series** have to be multiplied by data-set cardinality. For instance, if we have 1-million time-series and all individual time-series are below 4GB on disk we can expect Akumuli to use 20GB of RAM in worst case. It will actually use just above 10GB because of partial IOVec allocation. This is a terrific result since it will allow the database to store 10E15 data-points.

This won't work as good if you have a lot of small time-seires. For instance, if you have 10-million series and each one of those is small and fits 128KB around 40GB of RAM will be needed.

## Disk

Akumuli is designed for SSD and NVMe drives. It writes data sequentially and frees it in large blocks to avoid write amplification. All reads and writes are page alighned for the same purpose. Akumuli don't read anything from disk to write new data so queries can't deplete the read bandwidth and affect write speed. The database will work on HDD but it will work slower, especially on read side.

## CPU

Ingestion depend on number of available CPU's. In best case every CPU available for ingestion gives around 1M write/sec (if dictionary mode is used and on Intel CPU's). With OpenTSDB format the per-CPU write effeciency will get lower. CPU's can be provisioned for ingestion in configuration.