jjeong

- index settings 에서 number_of_shards 설정 정보를 이용합니다.

- 그래서 각 node 들에 순서데로 나눠서 할당을 합니다.

- sharding 된 index shard 에 document 를 어떻게 분배 할지 결정을 합니다.

- 분배는 hash 알고리즘을 이용합니다.

- Math.abs( hash(....) % numberOfShards )

{
    "query" : {
        "term" : {
            "field" : "value"
        }
    }
}

{
    "query" : {
        "filtered" : {
            "query" : {
                "match_all" : {}
            },
            "filter" : {
                "term" : {
                    "field" : "value"
                }
            }
        }
    }
}

{
    "query" : { … },
    "filter" : { … }
}

/{index}/_search
{
    "query" : { … },
    "size" : 0
}

/{index}/_search?search_type=count
{
    "query" : { … }
}

1. Query/filter to quickly find top N results
2. Rescore with complex logic to find top 10

_source.field

_fields.field

doc[field]

input {
    file {
        path => "/server/app/logstash/log/data.json"
        codec => json
        start_position => "beginning"
    }
}

output {
    stdout { debug => true debug_format => "json"}

    elasticsearch_http {
        host => "192.168.0.120"
        port => 9200
    }
}

input {
  file {
    add_field => ... # hash (optional), default: {}
    codec => ... # codec (optional), default: "plain"
    debug => ... # boolean (optional), default: false
    discover_interval => ... # number (optional), default: 15
    exclude => ... # array (optional)
    path => ... # array (required)
    sincedb_path => ... # string (optional)
    sincedb_write_interval => ... # number (optional), default: 15
    start_position => ... # string, one of ["beginning", "end"] (optional), default: "end"
    stat_interval => ... # number (optional), default: 1
    tags => ... # array (optional)
    type => ... # string (optional)
}

}

output {
  elasticsearch_http {
    codec => ... # codec (optional), default: "plain"
    document_id => ... # string (optional), default: nil
    flush_size => ... # number (optional), default: 100
    host => ... # string (required)
    idle_flush_time => ... # number (optional), default: 1
    index => ... # string (optional), default: "logstash-%{+YYYY.MM.dd}"
    index_type => ... # string (optional)
    manage_template => ... # boolean (optional), default: true
    password => ... # password (optional), default: nil
    port => ... # number (optional), default: 9200
    replication => ... # string, one of ["async", "sync"] (optional), default: "sync"
    template => ... # a valid filesystem path (optional)
    template_name => ... # string (optional), default: "logstash"
    template_overwrite => ... # boolean (optional), default: false
    user => ... # string (optional), default: nil
    workers => ... # number (optional), default: 1
}

}

Currently, we support file system, S3, Azure and HDFS repositories.

$ curator.py -h
usage: curator.py [-h] [-v] [--host HOST] [--port PORT] [-t TIMEOUT]
                  [-p PREFIX] [-s SEPARATOR] [-C CURATION_STYLE]
                  [-T TIME_UNIT] [-d DELETE_OLDER] [-c CLOSE_OLDER]
                  [-b BLOOM_OLDER] [-g DISK_SPACE]
                  [--max_num_segments MAX_NUM_SEGMENTS] [-o OPTIMIZE] [-n]
                  [-D] [-l LOG_FILE]

Curator for Elasticsearch indices. Can delete (by space or time), close,
disable bloom filters and optimize (forceMerge) your indices.

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program version number and exit
  --host HOST           Elasticsearch host. Default: localhost
  --port PORT           Elasticsearch port. Default: 9200
  -t TIMEOUT, --timeout TIMEOUT
                        Elasticsearch timeout. Default: 30
  -p PREFIX, --prefix PREFIX
                        Prefix for the indices. Indices that do not have this
                        prefix are skipped. Default: logstash-
  -s SEPARATOR, --separator SEPARATOR
                        Time unit separator. Default: .
  -C CURATION_STYLE, --curation-style CURATION_STYLE
                        Curate indices by [time, space] Default: time
  -T TIME_UNIT, --time-unit TIME_UNIT
                        Unit of time to reckon by: [days, hours] Default: days
  -d DELETE_OLDER, --delete DELETE_OLDER
                        Delete indices older than n TIME_UNITs.
  -c CLOSE_OLDER, --close CLOSE_OLDER
                        Close indices older than n TIME_UNITs.
  -b BLOOM_OLDER, --bloom BLOOM_OLDER
                        Disable bloom filter for indices older than n
                        TIME_UNITs.
  -g DISK_SPACE, --disk-space DISK_SPACE
                        Delete indices beyond n GIGABYTES.
  --max_num_segments MAX_NUM_SEGMENTS
                        Maximum number of segments, post-optimize. Default: 2
  -o OPTIMIZE, --optimize OPTIMIZE
                        Optimize (Lucene forceMerge) indices older than n
                        TIME_UNITs. Must increase timeout to stay connected
                        throughout optimize operation, recommend no less than
                        3600.
  -n, --dry-run         If true, does not perform any changes to the
                        Elasticsearch indices.
  -D, --debug           Debug mode
  -l LOG_FILE, --logfile LOG_FILE
                        log file

[[modules-discovery-zen]]
=== Zen Discovery

The zen discovery is the built in discovery module for elasticsearch and
the default. It provides both multicast and unicast discovery as well
being easily extended to support cloud environments.

▶ multicast, unicast, cloud 환경을 지원 하내요.

The zen discovery is integrated with other modules, for example, all
communication between nodes is done using the
<<modules-transport,transport>> module.

It is separated into several sub modules, which are explained below:

[float]
[[ping]]
==== Ping

This is the process where a node uses the discovery mechanisms to find
other nodes. There is support for both multicast and unicast based
discovery (can be used in conjunction as well).

▶ 노드를 찾거나 찔러 볼때 사용하내요.

[float]
[[multicast]]
===== Multicast
▶ 이건 사실 추천 하고 있지 않습니다. 불필요한 트래픽을 양산 한다고 해서요..
Multicast ping discovery of other nodes is done by sending one or more
multicast requests where existing nodes that exists will receive and
respond to. It provides the following settings with the
`discovery.zen.ping.multicast` prefix:

[cols="<,<",options="header",]
|=======================================================================
|Setting |Description
|`group` |The group address to use. Defaults to `224.2.2.4`.

|`port` |The port to use. Defaults to `54328`.

|`ttl` |The ttl of the multicast message. Defaults to `3`.

|`address` |The address to bind to, defaults to `null` which means it
will bind to all available network interfaces.
|=======================================================================

Multicast can be disabled by setting `multicast.enabled` to `false`.

[float]
[[unicast]]
===== Unicast
▶ 이걸 추천 하고 있죠.. ^^ 딱 보시면 아시겠지만 불특정 다수(?)에 보내는 multicast 방식과 다르게 지정한 노드로만 통신 하게 되니까 효율적이겠죠.
The unicast discovery allows to perform the discovery when multicast is
not enabled. It basically requires a list of hosts to use that will act
as gossip routers. It provides the following settings with the
`discovery.zen.ping.unicast` prefix:

[cols="<,<",options="header",]
|=======================================================================
|Setting |Description
|`hosts` |Either an array setting or a comma delimited setting. Each
value is either in the form of `host:port`, or in the form of
`host[port1-port2]`.
|=======================================================================

The unicast discovery uses the
<<modules-transport,transport>> module to
perform the discovery.

[float]
[[master-election]]
==== Master Election
▶ 이건 master node 가 장애가 났을 때 node.master: true 로 설정 되어 있는 노드들에서 선출 하는 기능 입니다.

▶ 전체 코디네이션 역할을 마스터 노드가 하기 떄문에 중요한 기능입니다.
As part of the initial ping process a master of the cluster is either
elected or joined to. This is done automatically. The
`discovery.zen.ping_timeout` (which defaults to `3s`) allows to
configure the election to handle cases of slow or congested networks
(higher values assure less chance of failure). Note, this setting was
changed from 0.15.1 onwards, prior it was called
`discovery.zen.initial_ping_timeout`.

Nodes can be excluded from becoming a master by setting `node.master` to
`false`. Note, once a node is a client node (`node.client` set to
`true`), it will not be allowed to become a master (`node.master` is
automatically set to `false`).

The `discovery.zen.minimum_master_nodes` allows to control the minimum
number of master eligible nodes a node should "see" in order to operate
within the cluster. Its recommended to set it to a higher value than 1
when running more than 2 nodes in the cluster.
▶ brain split 즉 네트웍 상에서 발생 가능한 단절로 인하여 데이터가 깨졌을 때를 방지 하기 위해 최소한 마스터 노드 역할을 수행 할 수 있는 노드를 2개 이상 설정 하라는 내용입니다.
[float]
[[fault-detection]]
==== Fault Detection
▶  이건 뭐 그냥 봐도 아시겠죠..
There are two fault detection processes running. The first is by the
master, to ping all the other nodes in the cluster and verify that they
are alive. And on the other end, each node pings to master to verify if
its still alive or an election process needs to be initiated.

The following settings control the fault detection process using the
`discovery.zen.fd` prefix:

[cols="<,<",options="header",]
|=======================================================================
|Setting |Description
|`ping_interval` |How often a node gets pinged. Defaults to `1s`.

|`ping_timeout` |How long to wait for a ping response, defaults to
`30s`.

|`ping_retries` |How many ping failures / timeouts cause a node to be
considered failed. Defaults to `3`.
|=======================================================================

[float]
==== External Multicast

The multicast discovery also supports external multicast requests to
discover nodes. The external client can send a request to the multicast
IP/group and port, in the form of:

[source,js]
--------------------------------------------------
{
    "request" : {
        "cluster_name": "test_cluster"
    }
}
--------------------------------------------------

And the response will be similar to node info response (with node level
information only, including transport/http addresses, and node
attributes):

[source,js]
--------------------------------------------------
{
    "response" : {
        "cluster_name" : "test_cluster",
        "transport_address" : "...",
        "http_address" : "...",
        "attributes" : {
            "..."
        }
    }
}
--------------------------------------------------

Note, it can still be enabled, with disabled internal multicast
discovery, but still have external discovery working by keeping
`discovery.zen.ping.multicast.enabled` set to `true` (the default), but,
setting `discovery.zen.ping.multicast.ping.enabled` to `false`.

#!/bin/bash

export ES_HEAP_SIZE=256m
export ES_HEAP_NEWSIZE=128m
export JAVA_OPT="-server -XX:+AggressiveOpts -XX:UseCompressedOops -XX:MaxDirectMemorySize -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly"

ES=/home/es/app/elasticsearch
$ES/bin/elasticsearch -p $ES/bin/es.pid -Des.config=$ES_NODE/config/elasticsearch.yml -Djava.net.preferIPv4Stack=true -Des.max-open-files=true > /dev/null 2>&1 &

#!/bin/bash

ES=/home/es/app/elasticsearch
/bin/kill `cat < $ES/bin/es.pid`

- number type index not_analyzed
{"type" : "long", "store" : "no", "index" : "not_analyzed", "index_options" : "docs", "ignore_malformed" : true, "include_in_all" : false}

- number type index no
{"type" : "long", "store" : "yes", "index" : "no", "ignore_malformed" : true, "include_in_all" : false}

- string type index analyzed
{"type" : "string", "store" : "no", "index" : "analyzed", "omit_norms" : false, "index_options" : "offsets", "term_vector" : "with_positions_offsets", "include_in_all" : false}

- string type index not_analyzed
{"type" : "string", "store" : "no", "index" : "not_analyzed", "omit_norms" : true, "index_options" : "docs", "include_in_all" : false}

- string type index no
{"type" : "string", "store" : "yes", "index" : "no", "include_in_all" : false}

- boolean type index yes
{"type" : "boolean", "store" : "yes", "include_in_all" : false}

- boolean type index no
{"type" : "boolean", "store" : "yes", "index" : "no", "include_in_all" : false}

Precision Step

You can choose any precisionStep when encoding values. Lower step values mean more precisions and so more terms in index (and index gets larger). The number of indexed terms per value is (those are generated by NumericTokenStream):

  indexedTermsPerValue = ceil(bitsPerValue / precisionStep)

As the lower precision terms are shared by many values, the additional terms only slightly grow the term dictionary (approx. 7% for precisionStep=4), but have a larger impact on the postings (the postings file will have more entries, as every document is linked to indexedTermsPerValue terms instead of one). The formula to estimate the growth of the term dictionary in comparison to one term per value:

On the other hand, if the precisionStep is smaller, the maximum number of terms to match reduces, which optimizes query speed. The formula to calculate the maximum number of terms that will be visited while executing the query is: