|
|
Elastic/Elasticsearch 2014. 1. 21. 16:09
elasticsearch 에서 클러스터 설정과 노드간의 통신에 대한 기본 정책을 설정 하는 역할을 discovery 라 보면 됩니다. 그럼 원문을 아래 살펴 볼까요? 원문 path : /elasticsearch/docs/reference/modules/discovery/zen.asciidoc ec2 참고 : /elasticsearch/docs/reference/modules/discovery/ec2.asciidoc
[[modules-discovery-zen]]
=== Zen Discovery
The zen discovery is the built in discovery module for elasticsearch and
the default. It provides both multicast and unicast discovery as well
being easily extended to support cloud environments. ▶ multicast, unicast, cloud 환경을 지원 하내요.
The zen discovery is integrated with other modules, for example, all
communication between nodes is done using the
<<modules-transport,transport>> module.
It is separated into several sub modules, which are explained below:
[float]
[[ping]]
==== Ping
This is the process where a node uses the discovery mechanisms to find
other nodes. There is support for both multicast and unicast based
discovery (can be used in conjunction as well). ▶ 노드를 찾거나 찔러 볼때 사용하내요.
[float]
[[multicast]]
===== Multicast
▶ 이건 사실 추천 하고 있지 않습니다. 불필요한 트래픽을 양산 한다고 해서요..
Multicast ping discovery of other nodes is done by sending one or more
multicast requests where existing nodes that exists will receive and
respond to. It provides the following settings with the
`discovery.zen.ping.multicast` prefix:
[cols="<,<",options="header",]
|=======================================================================
|Setting |Description
|`group` |The group address to use. Defaults to `224.2.2.4`.
|`port` |The port to use. Defaults to `54328`.
|`ttl` |The ttl of the multicast message. Defaults to `3`.
|`address` |The address to bind to, defaults to `null` which means it
will bind to all available network interfaces.
|=======================================================================
Multicast can be disabled by setting `multicast.enabled` to `false`.
[float]
[[unicast]]
===== Unicast
▶ 이걸 추천 하고 있죠.. ^^ 딱 보시면 아시겠지만 불특정 다수(?)에 보내는 multicast 방식과 다르게 지정한 노드로만 통신 하게 되니까 효율적이겠죠.
The unicast discovery allows to perform the discovery when multicast is
not enabled. It basically requires a list of hosts to use that will act
as gossip routers. It provides the following settings with the
`discovery.zen.ping.unicast` prefix:
[cols="<,<",options="header",]
|=======================================================================
|Setting |Description
|`hosts` |Either an array setting or a comma delimited setting. Each
value is either in the form of `host:port`, or in the form of
`host[port1-port2]`.
|=======================================================================
The unicast discovery uses the
<<modules-transport,transport>> module to
perform the discovery.
[float]
[[master-election]]
==== Master Election
▶ 이건 master node 가 장애가 났을 때 node.master: true 로 설정 되어 있는 노드들에서 선출 하는 기능 입니다.
▶ 전체 코디네이션 역할을 마스터 노드가 하기 떄문에 중요한 기능입니다.
As part of the initial ping process a master of the cluster is either
elected or joined to. This is done automatically. The
`discovery.zen.ping_timeout` (which defaults to `3s`) allows to
configure the election to handle cases of slow or congested networks
(higher values assure less chance of failure). Note, this setting was
changed from 0.15.1 onwards, prior it was called
`discovery.zen.initial_ping_timeout`.
Nodes can be excluded from becoming a master by setting `node.master` to
`false`. Note, once a node is a client node (`node.client` set to
`true`), it will not be allowed to become a master (`node.master` is
automatically set to `false`).
The `discovery.zen.minimum_master_nodes` allows to control the minimum
number of master eligible nodes a node should "see" in order to operate
within the cluster. Its recommended to set it to a higher value than 1
when running more than 2 nodes in the cluster.
▶ brain split 즉 네트웍 상에서 발생 가능한 단절로 인하여 데이터가 깨졌을 때를 방지 하기 위해 최소한 마스터 노드 역할을 수행 할 수 있는 노드를 2개 이상 설정 하라는 내용입니다.
[float]
[[fault-detection]]
==== Fault Detection
▶ 이건 뭐 그냥 봐도 아시겠죠..
There are two fault detection processes running. The first is by the
master, to ping all the other nodes in the cluster and verify that they
are alive. And on the other end, each node pings to master to verify if
its still alive or an election process needs to be initiated.
The following settings control the fault detection process using the
`discovery.zen.fd` prefix:
[cols="<,<",options="header",]
|=======================================================================
|Setting |Description
|`ping_interval` |How often a node gets pinged. Defaults to `1s`.
|`ping_timeout` |How long to wait for a ping response, defaults to
`30s`.
|`ping_retries` |How many ping failures / timeouts cause a node to be
considered failed. Defaults to `3`.
|=======================================================================
[float]
==== External Multicast
The multicast discovery also supports external multicast requests to
discover nodes. The external client can send a request to the multicast
IP/group and port, in the form of:
[source,js]
--------------------------------------------------
{
"request" : {
"cluster_name": "test_cluster"
}
}
--------------------------------------------------
And the response will be similar to node info response (with node level
information only, including transport/http addresses, and node
attributes):
[source,js]
--------------------------------------------------
{
"response" : {
"cluster_name" : "test_cluster",
"transport_address" : "...",
"http_address" : "...",
"attributes" : {
"..."
}
}
}
--------------------------------------------------
Note, it can still be enabled, with disabled internal multicast
discovery, but still have external discovery working by keeping
`discovery.zen.ping.multicast.enabled` set to `true` (the default), but,
setting `discovery.zen.ping.multicast.ping.enabled` to `false`.
이상 살펴 봤습니다. 뭐 별거 없죠.. ^^
[세 줄 요약] 1. unicast 를 사용해라. 2. multicast 는 disable 해라. 3. minimum master node 는 2개 이상 설정해라.
Elastic/Elasticsearch 2014. 1. 17. 10:55
분명 글을 작성했던 것 같은데 보이질 않내요. 그래서 다시 올려 봅니다. ㅡ.ㅡ;;
[start.sh] #!/bin/bash
export ES_HEAP_SIZE=256m
export ES_HEAP_NEWSIZE=128m
export JAVA_OPT="-server -XX:+AggressiveOpts -XX:UseCompressedOops -XX:MaxDirectMemorySize -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly"
ES=/home/es/app/elasticsearch
$ES/bin/elasticsearch -p $ES/bin/es.pid -Des.config=$ES_NODE/config/elasticsearch.yml -Djava.net.preferIPv4Stack=true -Des.max-open-files=true > /dev/null 2>&1 &
[stop.sh] #!/bin/bash
ES=/home/es/app/elasticsearch
/bin/kill `cat < $ES/bin/es.pid`
Elastic/Elasticsearch 2014. 1. 14. 17:34
참고글 : http://stackoverflow.com/questions/15019821/what-differents-between-master-node-gateway-and-other-node-gateway-in-elasticsea 참고 하시라고 올려 봅니다.
[원문]
The master node is the same as any other node in the cluster, except that it has been elected to be the master. It is responsible for coordinating any cluster-wide changes, such as
as the addition or removal of a node, creation, deletion or change of
state (ie open/close) of an index, and the allocation of shards to
nodes. When any of these changes occur, the "cluster state" is updated
by the master and published to all other nodes in the cluster. It is the
only node that may publish a new cluster state. The tasks that a master performs are lightweight. Any tasks that deal
with data (eg indexing, searching etc) do not need to involve the
master. If you choose to run the master as a non-data node (ie a node
that acts as master and as a router, but doesn't contain any data) then
the master can run happily on a smallish box. A node is allowed to become a master if it is marked as "master
eligible" (which all nodes are by default). If the current master goes
down, a new master will be elected by the cluster. An important configuration option in your cluster is minimum_master_nodes.
This specifies the number of "master eligible" nodes that a node must
be able to see in order to be part of a cluster. Its purpose is to
avoid "split brain" ie having the cluster separate into two clusters,
both of which think that they are functioning correctly. For instance, if you have 3 nodes, all of which are master eligible, and set minimum_master_nodes
to 1, then if the third node is separated from the other two it, it
still sees one master-eligible node (itself) and thinks that it can form
a cluster by itself. Instead, set minimum_master_nodes to 2 in this case
(number of nodes / 2 + 1), then if the third node separates, it won't
see enough master nodes, and thus won't form a cluster by itself. It
will keep trying to join the original cluster. While Elasticsearch tries very hard to choose the correct defaults, minimum_master_nodes
is impossible to guess, as it has no way of knowing how many nodes you
intend to run. This is something you must configure yourself.
[구글 번역] 마스터 노드 는마스터로 선출 되었음을 제외하고 ,클러스터의 다른 노드 와 동일하다.
Elastic/Elasticsearch 2014. 1. 10. 17:53
공유했다고 생각했는데 아니였나 보내요. 이미 많은 분들이 알고 계실수도 있지만, 반복학습 차원에서.. ^^
https://github.com/bleskes/sense
Sense
A JSON aware developer's interface to ElasticSearch. Comes with handy machinery such as syntax highlighting, autocomplete,
formatting and code folding.
Installation
Sense is installed as a Chrome Extension. Install it from
the Chrome Webstore .
Elastic/Elasticsearch 2014. 1. 8. 18:33
아래 글 참고 http://jjeong.tistory.com/914
이해하기 쉽도록 힌트 몇자 적습니다. 기본 이해는
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html 이 문서를 보시면 됩니다.
step 1. template 을 생성 합니다. curl -XPUT localhost:9200/_template/template_1 -d '
{
"template" : "te*",
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"type1" : {
"_source" : { "enabled" : false }
}
}
}
' 여기서 template : te* 이 의미 하는 것은 index 명입니다.
step 2.
curl -XPUT 'http://localhost:9200/temp/' 이렇게 생성하면 temp 라는 인덱스의 settings/mappings 정보는 template_1 값을 가지게 됩니다.
logstash 예제로 보겠습니다. 아래는 template 생성용 json 입니다. { "template" : "logstash-*", "settings" : { "index.refresh_interval" : "5s", "analysis" : { "analyzer" : { "default" : { "type" : "standard", "stopwords" : "_none_" } } } }, "mappings" : { "_default_" : { "_all" : {"enabled" : true}, "dynamic_templates" : [ { "string_fields" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "type" : "multi_field", "fields" : { "{name}" : {"type": "string", "index" : "analyzed", "omit_norms" : true, "index_options" : "docs"}, "{name}.raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256} } } } } ], "properties" : { "@version": { "type": "string", "index": "not_analyzed" }, "geoip" : { "type" : "object", "dynamic": true, "path": "full", "properties" : { "location" : { "type" : "geo_point" } } } } } } }
보시면 인덱스 명이 logstash-* 로 시작하는 것들은 이 템플릿을 따르게 됩니다. _all 을 enable 한 이유는 특정 필드에 대해서 동적으로 검색을 지원하기 위해서 라고 보시면 됩니다. 특히 string 필드에 대해서는 검색을 하는 것으로 지정을 하였고, multi_field 구성한 이유는 not_analyzed 로 봐서는 facet 기능이나 sort 등의 다른 기능을 활용하기 위해서 인것으로 보입니다.
그럼 이만... :)
Elastic/Elasticsearch 2014. 1. 8. 12:08
색인 스키마를 json 파일로 만들어 놓고 rest api 로 생성 할 때 유용한 스크립트 입니다. 그냥 제가 사용하기 편할라고 대충 만들어 놓은거랍니다.
#!/bin/bash
size=$#
if [ $size -ne 3 ]; then
echo "Usage: create_index.sh IP:PORT INDICE SCHEME_FILE";
echo "Example: create_index.sh localhost:9200 idx_local schema.json";
exit 0;
fi
serviceUri=$1
indice=$2
schema=$3
curl -XDELETE 'http://'$serviceUri'/'$indice
curl -XPUT 'http://'$serviceUri'/'$indice -d @$schema
Elastic/Elasticsearch 2014. 1. 7. 18:55
색인 스키마 관리를 위해서 템플릿 생성을 할 수 있습니다. 쉽게 접할 수 있는 예제로 logstash 정보가 괜찮아 보여서 공유합니다.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-root-object-type.html#_dynamic_templates
https://gist.github.com/deverton/2970285 https://github.com/logstash/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json
{ "template": "logstash-*", "settings" : { "number_of_shards" : 1, "number_of_replicas" : 0, "index" : { "query" : { "default_field" : "@message" }, "store" : { "compress" : { "stored" : true, "tv": true } } } }, "mappings": { "_default_": { "_all": { "enabled": false }, "_source": { "compress": true }, "dynamic_templates": [ { "string_template" : { "match" : "*", "mapping": { "type": "string", "index": "not_analyzed" }, "match_mapping_type" : "string" } } ], "properties" : { "@fields": { "type": "object", "dynamic": true, "path": "full" }, "@message" : { "type" : "string", "index" : "analyzed" }, "@source" : { "type" : "string", "index" : "not_analyzed" }, "@source_host" : { "type" : "string", "index" : "not_analyzed" }, "@source_path" : { "type" : "string", "index" : "not_analyzed" }, "@tags": { "type": "string", "index" : "not_analyzed" }, "@timestamp" : { "type" : "date", "index" : "not_analyzed" }, "@type" : { "type" : "string", "index" : "not_analyzed" } } } } }
{ "template" : "logstash-*", "settings" : { "index.refresh_interval" : "5s", "analysis" : { "analyzer" : { "default" : { "type" : "standard", "stopwords" : "_none_" } } } }, "mappings" : { "_default_" : { "_all" : {"enabled" : true}, "dynamic_templates" : [ { "string_fields" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "type" : "multi_field", "fields" : { "{name}" : {"type": "string", "index" : "analyzed", "omit_norms" : true, "index_options" : "docs"}, "{name}.raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256} } } } } ], "properties" : { "@version": { "type": "string", "index": "not_analyzed" }, "geoip" : { "type" : "object", "dynamic": true, "path": "full", "properties" : { "location" : { "type" : "geo_point" } } } } } } }
Elastic/Elasticsearch 2014. 1. 7. 18:41
그냥 참고용으로 올려 놓는 것입니다. 각 속성들은 서비스 특성에 맞춰서 설정 하시는게 좋습니다.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-put-mapping.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html
{
"settings" : {
"number_of_shards" : 5,
"number_of_replicas" : 0,
"index" : {
"refresh_interval" : "1s",
"merge" : {
"policy" : { "segments_per_tier" : 5 }
},
"analysis" : {
"analyzer" : {
"analyzer_standard" : {
"type" : "standard",
"tokenizer" : "whitespace",
"filter" : ["lowercase", "trim"]
},
"analyzer_pattern" : {
"type" : "custom",
"tokenizer" : "tokenizer_pattern",
"filter" : ["lowercase", "trim"]
},
"analyzer_ngram" : {
"type" : "custom",
"tokenizer" : "tokenizer_ngram",
"filter" : ["lowercase", "trim"]
}
},
"tokenizer" : {
"tokenizer_ngram" : {
"type" : "nGram",
"min_gram" : "2",
"max_gram" : "10",
"token_chars": [ "letter", "digit" ]
},
"tokenizer_pattern" : {
"type" : "pattern",
"pattern" : ","
}
}
},
"store" : {
"type" : "mmapfs",
"compress" : {
"stored" : true,
"tv" : true
}
}
}
},
"mappings" : {
"INDICE_TYPE_NAME" : {
"_id" : {
"index" : "not_analyzed",
"path" : "KEY_FIELD_NAME"
},
"_source" : {
"enabled" : "true"
},
"_all" : {
"enabled" : "false"
},
"_boost" : {
"name" : "_boost",
"null_value" : 1.0
},
"analyzer" : "analyzer_standard",
"index_analyzer" : "analyzer_standard",
"search_analyzer" : "analyzer_standard",
"properties" : {
"LONG_KEY_FIELD" : {"type" : "long", "store" : "no", "index" : "not_analyzed", "omit_norms" : true, "index_options" : "docs", "ignore_malformed" : true, "include_in_all" : false},
"STRING_SEARCH_FIELD" : {"type" : "string", "store" : "no", "index" : "analyzed", "omit_norms" : false, "index_options" : "offsets", "term_vector" : "with_positions_offsets", "include_in_all" : false},
"STRING_VIEW_FIELD" : {"type" : "string", "store" : "yes", "index" : "no", "include_in_all" : false},
"INTEGER_KEY_FIELD" : {"type" : "integer", "store" : "no", "index" : "not_analyzed", "omit_norms" : true, "index_options" : "docs", "ignore_malformed" : true, "include_in_all" : false},
"FLOAT_KEY_FIELD" : {"type" : "float", "store" : "no", "index" : "not_analyzed", "omit_norms" : true, "index_options" : "docs", "ignore_malformed" : true, "include_in_all" : false},
"LONG_VIEW_FIELD" : {"type" : "long", "store" : "yes", "index" : "no", "ignore_malformed" : true, "include_in_all" : false},
"STRING_KEY_FIELD" : {"type" : "string", "store" : "no", "index" : "not_analyzed", "omit_norms" : true, "index_options" : "docs", "include_in_all" : false},
"NESTED_KEY_FIELD" : {"type" : "nested",
"properties" : {
"STRING_KEY_FIELD" : {"type" : "string", "store" : "no", "index" : "not_analyzed", "omit_norms" : true, "index_options" : "docs", "include_in_all" : false},
"INTEGER_VIEW_FIELD" : {"type" : "integer", "store" : "yes", "index" : "no", "ignore_malformed" : true, "include_in_all" : false}
}
},
"BOOLEAN_VIEW_FIELD" : {"type" : "boolean", "store" : "yes", "include_in_all" : false},
"BOOLEAN_KEY_FIELD" : {"type" : "boolean", "store" : "no", "index" : "not_analyzed", "omit_norms" : true, "index_options" : "docs", "include_in_all" : false},
"OBJECT_VIEW_FIELD" : {"type" : "object", "dynamic" : true, "store" : "yes", "index" : "no", "include_in_all" : false}
}
}
}
}
Elastic/Elasticsearch 2014. 1. 7. 12:30
http://www.elasticsearch.org/contributing-to-elasticsearch/
elasticsearch checkout 받은 후 코드 수정 또는 빌드를 하고 싶을때 참고 하시면 됩니다. 이전 버전에서는 maven 2.x 에서 되었던 것 같은데 지금은 3.x 가 필요 하내요. run configure 에서 maven 3.x 로 변경 하시고 빌드 하시면 정상적으로 빌드가 됩니다.
위 문서에 나와 있는 것 처럼 Goals 에 아래 옵션을 넣고 빌드 하세요. clean package -DskipTests
Elastic/Elasticsearch 2013. 12. 16. 17:04
간혹 elasticsearch 가 비정상 종료 되거나 했을 경우 unassigned shard 가 남는 경우가 있습니다. 이럴 경우 아래와 같이 재할당을 할 수 있는데요. 운영 하실 때 참고 하면 좋습니다.
[shard_reroute.sh]
#!/bin/bash
size=$#
if [ $size -ne 2 ]; then
echo "Usage: shard.sh INDICE_NAME SHARD_NUMBER";
echo "Example: shard.sh products 1";
exit 0;
fi
indice=$1
shard=$2
echo "curl -XPOST http://localhost:9200/_cluster/reroute -d '{ "commands" : [ { "allocate" : { "index" : \"$indice\", "shard" : $shard, "node" : "node1", "allow_primary" : true } } ] }'"
echo ""
curl -XPOST http://localhost:9200/_cluster/reroute -d '{ "commands" : [ { "allocate" : { "index" : "'$indice'", "shard" : '$shard', "node" : "node1", "allow_primary" : true } } ] }'
[Reference URL] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-reroute.html
|
