|
|
Elastic/Elasticsearch 2017. 1. 2. 12:44
도대체 왜 맨날 잊어버리는지 모르겠지만, 기억력 회복을 위해 기록해 봅니다.
Range query 사용 시 from, to, gt, gte, lt, lte parameter 를 사용 합니다. RangeQueryBuilder.java 소스코드를 보면 아래와 같이 정의가 되어 있습니다.
private final String name;
private Object from;
private Object to;
private String timeZone;
private boolean includeLower = true;
private boolean includeUpper = true;
private float boost = -1;
private String queryName;
private String format;
기본적으로 lower, upper 값을 포함하게 되어 있습니다. 그러므로, from, to 는 값을 포함 하게 됩니다. MySQL 에서 제공하고 있는 BETWEEN min AND max 도 min 과 max 값을 포함 하고 있는 것 처럼 동일 합니다.
Elastic/Elasticsearch 2016. 12. 6. 12:18
그냥 볼일이 있어서 올려봅니다. nodes 랑 indices 만.
[참고문서] https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-nodes.html id | id,nodeId | unique node id
pid | p | process id
host | h | host name
ip | i | ip address
port | po | bound transport port
version | v | es version
build | b | es build hash
jdk | j | jdk version
disk.avail | d,disk,diskAvail | available disk space
heap.current | hc,heapCurrent | used heap
heap.percent | hp,heapPercent | used heap ratio
heap.max | hm,heapMax | max configured heap
ram.current | rc,ramCurrent | used machine memory
ram.percent | rp,ramPercent | used machine memory ratio
ram.max | rm,ramMax | total machine memory
file_desc.current | fdc,fileDescriptorCurrent | used file descriptors
file_desc.percent | fdp,fileDescriptorPercent | used file descriptor ratio
file_desc.max | fdm,fileDescriptorMax | max file descriptors
cpu | cpu | recent cpu usage
load | l | most recent load avg
uptime | u | node uptime
node.role | r,role,dc,nodeRole | d:data node, c:client node
master | m | m:master-eligible, *:current master
name | n | node name
completion.size | cs,completionSize | size of completion
fielddata.memory_size | fm,fielddataMemory | used fielddata cache
fielddata.evictions | fe,fielddataEvictions | fielddata evictions
query_cache.memory_size | qcm,queryCacheMemory | used query cache
query_cache.evictions | qce,queryCacheEvictions | query cache evictions
request_cache.memory_size | rcm,requestCacheMemory | used request cache
request_cache.evictions | rce,requestCacheEvictions | request cache evictions
request_cache.hit_count | rchc,requestCacheHitCount | request cache hit counts
request_cache.miss_count | rcmc,requestCacheMissCount | request cache miss counts
flush.total | ft,flushTotal | number of flushes
flush.total_time | ftt,flushTotalTime | time spent in flush
get.current | gc,getCurrent | number of current get ops
get.time | gti,getTime | time spent in get
get.total | gto,getTotal | number of get ops
get.exists_time | geti,getExistsTime | time spent in successful gets
get.exists_total | geto,getExistsTotal | number of successful gets
get.missing_time | gmti,getMissingTime | time spent in failed gets
get.missing_total | gmto,getMissingTotal | number of failed gets
indexing.delete_current | idc,indexingDeleteCurrent | number of current deletions
indexing.delete_time | idti,indexingDeleteTime | time spent in deletions
indexing.delete_total | idto,indexingDeleteTotal | number of delete ops
indexing.index_current | iic,indexingIndexCurrent | number of current indexing ops
indexing.index_time | iiti,indexingIndexTime | time spent in indexing
indexing.index_total | iito,indexingIndexTotal | number of indexing ops
indexing.index_failed | iif,indexingIndexFailed | number of failed indexing ops
merges.current | mc,mergesCurrent | number of current merges
merges.current_docs | mcd,mergesCurrentDocs | number of current merging docs
merges.current_size | mcs,mergesCurrentSize | size of current merges
merges.total | mt,mergesTotal | number of completed merge ops
merges.total_docs | mtd,mergesTotalDocs | docs merged
merges.total_size | mts,mergesTotalSize | size merged
merges.total_time | mtt,mergesTotalTime | time spent in merges
percolate.current | pc,percolateCurrent | number of current percolations
percolate.memory_size | pm,percolateMemory | memory used by percolations
percolate.queries | pq,percolateQueries | number of registered percolation queries
percolate.time | pti,percolateTime | time spent percolating
percolate.total | pto,percolateTotal | total percolations
refresh.total | rto,refreshTotal | total refreshes
refresh.time | rti,refreshTime | time spent in refreshes
script.compilations | scrcc,scriptCompilations | script compilations
script.cache_evictions | scrce,scriptCacheEvictions | script cache evictions
search.fetch_current | sfc,searchFetchCurrent | current fetch phase ops
search.fetch_time | sfti,searchFetchTime | time spent in fetch phase
search.fetch_total | sfto,searchFetchTotal | total fetch ops
search.open_contexts | so,searchOpenContexts | open search contexts
search.query_current | sqc,searchQueryCurrent | current query phase ops
search.query_time | sqti,searchQueryTime | time spent in query phase
search.query_total | sqto,searchQueryTotal | total query phase ops
search.scroll_current | scc,searchScrollCurrent | open scroll contexts
search.scroll_time | scti,searchScrollTime | time scroll contexts held open
search.scroll_total | scto,searchScrollTotal | completed scroll contexts
segments.count | sc,segmentsCount | number of segments
segments.memory | sm,segmentsMemory | memory used by segments
segments.index_writer_memory | siwm,segmentsIndexWriterMemory | memory used by index writer
segments.index_writer_max_memory | siwmx,segmentsIndexWriterMaxMemory | maximum memory index writer may use before it must write buffered documents to a new segment
segments.version_map_memory | svmm,segmentsVersionMapMemory | memory used by version map
segments.fixed_bitset_memory | sfbm,fixedBitsetMemory | memory used by fixed bit sets for nested object field types and type filters for types referred in _parent fields
suggest.current | suc,suggestCurrent | number of current suggest ops
suggest.time | suti,suggestTime | time spend in suggest
suggest.total | suto,suggestTotal | number of suggest ops
[참고문서] https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html health | h | current health status
status | s | open/close status
index | i,idx | index name
pri | p,shards.primary,shardsPrimary | number of primary shards
rep | r,shards.replica,shardsReplica | number of replica shards
docs.count | dc,docsCount | available docs
docs.deleted | dd,docsDeleted | deleted docs
creation.date | cd | index creation date (millisecond value)
creation.date.string | cds | index creation date (as string)
store.size | ss,storeSize | store size of primaries & replicas
pri.store.size | | store size of primaries
completion.size | cs,completionSize | size of completion
pri.completion.size | | size of completion
fielddata.memory_size | fm,fielddataMemory | used fielddata cache
pri.fielddata.memory_size | | used fielddata cache
fielddata.evictions | fe,fielddataEvictions | fielddata evictions
pri.fielddata.evictions | | fielddata evictions
query_cache.memory_size | qcm,queryCacheMemory | used query cache
pri.query_cache.memory_size | | used query cache
query_cache.evictions | qce,queryCacheEvictions | query cache evictions
pri.query_cache.evictions | | query cache evictions
request_cache.memory_size | rcm,requestCacheMemory | used request cache
pri.request_cache.memory_size | | used request cache
request_cache.evictions | rce,requestCacheEvictions | request cache evictions
pri.request_cache.evictions | | request cache evictions
request_cache.hit_count | rchc,requestCacheHitCount | request cache hit count
pri.request_cache.hit_count | | request cache hit count
request_cache.miss_count | rcmc,requestCacheMissCount | request cache miss count
pri.request_cache.miss_count | | request cache miss count
flush.total | ft,flushTotal | number of flushes
pri.flush.total | | number of flushes
flush.total_time | ftt,flushTotalTime | time spent in flush
pri.flush.total_time | | time spent in flush
get.current | gc,getCurrent | number of current get ops
pri.get.current | | number of current get ops
get.time | gti,getTime | time spent in get
pri.get.time | | time spent in get
get.total | gto,getTotal | number of get ops
pri.get.total | | number of get ops
get.exists_time | geti,getExistsTime | time spent in successful gets
pri.get.exists_time | | time spent in successful gets
get.exists_total | geto,getExistsTotal | number of successful gets
pri.get.exists_total | | number of successful gets
get.missing_time | gmti,getMissingTime | time spent in failed gets
pri.get.missing_time | | time spent in failed gets
get.missing_total | gmto,getMissingTotal | number of failed gets
pri.get.missing_total | | number of failed gets
indexing.delete_current | idc,indexingDeleteCurrent | number of current deletions
pri.indexing.delete_current | | number of current deletions
indexing.delete_time | idti,indexingDeleteTime | time spent in deletions
pri.indexing.delete_time | | time spent in deletions
indexing.delete_total | idto,indexingDeleteTotal | number of delete ops
pri.indexing.delete_total | | number of delete ops
indexing.index_current | iic,indexingIndexCurrent | number of current indexing ops
pri.indexing.index_current | | number of current indexing ops
indexing.index_time | iiti,indexingIndexTime | time spent in indexing
pri.indexing.index_time | | time spent in indexing
indexing.index_total | iito,indexingIndexTotal | number of indexing ops
pri.indexing.index_total | | number of indexing ops
indexing.index_failed | iif,indexingIndexFailed | number of failed indexing ops
pri.indexing.index_failed | | number of failed indexing ops
merges.current | mc,mergesCurrent | number of current merges
pri.merges.current | | number of current merges
merges.current_docs | mcd,mergesCurrentDocs | number of current merging docs
pri.merges.current_docs | | number of current merging docs
merges.current_size | mcs,mergesCurrentSize | size of current merges
pri.merges.current_size | | size of current merges
merges.total | mt,mergesTotal | number of completed merge ops
pri.merges.total | | number of completed merge ops
merges.total_docs | mtd,mergesTotalDocs | docs merged
pri.merges.total_docs | | docs merged
merges.total_size | mts,mergesTotalSize | size merged
pri.merges.total_size | | size merged
merges.total_time | mtt,mergesTotalTime | time spent in merges
pri.merges.total_time | | time spent in merges
percolate.current | pc,percolateCurrent | number of current percolations
pri.percolate.current | | number of current percolations
percolate.memory_size | pm,percolateMemory | memory used by percolations
pri.percolate.memory_size | | memory used by percolations
percolate.queries | pq,percolateQueries | number of registered percolation queries
pri.percolate.queries | | number of registered percolation queries
percolate.time | pti,percolateTime | time spent percolating
pri.percolate.time | | time spent percolating
percolate.total | pto,percolateTotal | total percolations
pri.percolate.total | | total percolations
refresh.total | rto,refreshTotal | total refreshes
pri.refresh.total | | total refreshes
refresh.time | rti,refreshTime | time spent in refreshes
pri.refresh.time | | time spent in refreshes
search.fetch_current | sfc,searchFetchCurrent | current fetch phase ops
pri.search.fetch_current | | current fetch phase ops
search.fetch_time | sfti,searchFetchTime | time spent in fetch phase
pri.search.fetch_time | | time spent in fetch phase
search.fetch_total | sfto,searchFetchTotal | total fetch ops
pri.search.fetch_total | | total fetch ops
search.open_contexts | so,searchOpenContexts | open search contexts
pri.search.open_contexts | | open search contexts
search.query_current | sqc,searchQueryCurrent | current query phase ops
pri.search.query_current | | current query phase ops
search.query_time | sqti,searchQueryTime | time spent in query phase
pri.search.query_time | | time spent in query phase
search.query_total | sqto,searchQueryTotal | total query phase ops
pri.search.query_total | | total query phase ops
search.scroll_current | scc,searchScrollCurrent | open scroll contexts
pri.search.scroll_current | | open scroll contexts
search.scroll_time | scti,searchScrollTime | time scroll contexts held open
pri.search.scroll_time | | time scroll contexts held open
search.scroll_total | scto,searchScrollTotal | completed scroll contexts
pri.search.scroll_total | | completed scroll contexts
segments.count | sc,segmentsCount | number of segments
pri.segments.count | | number of segments
segments.memory | sm,segmentsMemory | memory used by segments
pri.segments.memory | | memory used by segments
segments.index_writer_memory | siwm,segmentsIndexWriterMemory | memory used by index writer
pri.segments.index_writer_memory | | memory used by index writer
segments.index_writer_max_memory | siwmx,segmentsIndexWriterMaxMemory | maximum memory index writer may use before it must write buffered documents to a new segment
pri.segments.index_writer_max_memory | | maximum memory index writer may use before it must write buffered documents to a new segment
segments.version_map_memory | svmm,segmentsVersionMapMemory | memory used by version map
pri.segments.version_map_memory | | memory used by version map
segments.fixed_bitset_memory | sfbm,fixedBitsetMemory | memory used by fixed bit sets for nested object field types and type filters for types referred in _parent fields
pri.segments.fixed_bitset_memory | | memory used by fixed bit sets for nested object field types and type filters for types referred in _parent fields
warmer.current | wc,warmerCurrent | current warmer ops
pri.warmer.current | | current warmer ops
warmer.total | wto,warmerTotal | total warmer ops
pri.warmer.total | | total warmer ops
warmer.total_time | wtt,warmerTotalTime | time spent in warmers
pri.warmer.total_time | | time spent in warmers
suggest.current | suc,suggestCurrent | number of current suggest ops
pri.suggest.current | | number of current suggest ops
suggest.time | suti,suggestTime | time spend in suggest
pri.suggest.time | | time spend in suggest
suggest.total | suto,suggestTotal | number of suggest ops
pri.suggest.total | | number of suggest ops
memory.total | tm,memoryTotal | total used memory
pri.memory.total | | total user memory
Elastic/Elasticsearch 2016. 11. 25. 12:31
Elasticsearch cluster 업그레이드를 위해 먼저 한글형태소 분석기 업그레이드가 필요합니다. 기본적으로 한글형태소 분석기 플러그인을 만들기 위해서는 아래의 내용을 어느 정도는 잘 알고 다룰수 있어야 합니다.
- Elasticsearch - Lucene - Arirang
Arirang 은 아래 링크를 통해서 소스와 jar 파일을 구하실 수 있습니다.
최근에 수명님 이외 mgkaki 님이 컨트리뷰션을 해주시고 계신듯 합니다. :)
Lucene & Arirang 변경 사항) - lucene 6.1 과 6.2 의 패키지 구조가 변경이 되고 클래스도 바뀌었습니다. - arirang 에서 제공하던 pairmap 관련 버그가 수정되었습니다. (그전에 수정이 되었을수도 있습니다. ^^;) - lucene 에서 제공 되던 CharacterUtils 가 refactoring 되었습니다. - arirang 에서 KoreanTokenizer 에 선언된 CharacterUtils 를 변경된 내용에 맞게 고쳐주어야 합니다.
Remove CharacterUtils.getInstance() CharacterUtils.codePointAt(...) to Character.codePointAt(...)
- arirang 6.2 source를 내려 받으시면 위 변경 내용이 반영 되어 있습니다. - arirang.morph 1.1.0 을 내려 받으셔야 합니다.
Elasticsearch Plugin 변경 사항) 플러그인 개발 변경 사항은 기본 구조 변경이 많이 되었기 때문에 수정 사항이 많습니다. 보기에 따라서 적을 수도 있지만 판단은 각자의 몫으로 ^^
- arirang.lucene-analyzer 와 arirang-morph 업데이트가 되어야 합니다. - 기존에 binding 하던 AnalysisBinderProcessor를 사용하지 않습니다. - 이제는 Plugin, AnalysisPlugin 에서 등록을 진행 합니다.
public class AnalysisArirangPlugin extends Plugin implements AnalysisPlugin { @Override public Map<String, AnalysisProvider<TokenFilterFactory>> getTokenFilters() { return singletonMap("arirang_filter", ArirangTokenFilterFactory::new); }
@Override public Map<String, AnalysisProvider<TokenizerFactory>> getTokenizers() { Map<String, AnalysisProvider<TokenizerFactory>> extra = new HashMap<>(); extra.put("arirang_tokenizer", ArirangTokenizerFactory::new);
return extra; }
@Override public Map<String, AnalysisProvider<AnalyzerProvider<? extends Analyzer>>> getAnalyzers() { return singletonMap("arirang_analyzer", ArirangAnalyzerProvider::new); } }
- AnalyzerProvider, TokenFilterFactory, TokenizerFactory 내 생성자 argument 가 바뀌었습니다. IndexSettings indexSettings, Environment env, String name, Settings settings
- assemble 하기 위한 plugin.xml 내 outputDirectory 가 elasticsearch 로 변경이 되었습니다. - outputDirectory 가 elasticsearch 로 작성되어 있지 않을 경우 에러가 발생 합니다.
이 정도 변경 하고 나면 이제 빌드 및 설치를 하셔도 됩니다. 이전 글 참고) [Elasticsearch] Lucene Arirang Analyzer Plugin for Elasticsearch 5.0.1
※ 플러그인을 만들면서 우선 lucene 6.1 과 6.2 가 바뀌어서 살짝 당황 했었습니다. 당연히 6.x 간에는 패키지 구조에 대한 변경은 없을거라는 기대를 했었는데 이게 잘못이였던 것 같습니다.
역시 lucene 5.x 에서 6.x 로 넘어 가기 때문에 elasticsearch 5.x 는 많이 바뀌었을 거라는 생각은 했었구요. 그래도 생각했던 것 보다 오래 걸리지는 않았지만 역시 참고할 만한 문서나 자료는 어디에도 없더라구요. 소스 보는게 진리라는건 변하지 않는 듯 싶내요. 작성하고 보니 이게 개발기인지 애매하내요. ^^;
소스코드) https://github.com/HowookJeong/elasticsearch-analysis-arirang
Elastic/Elasticsearch 2016. 11. 24. 19:02
우선 빌드한 플러그인 zip 파일 먼저 공유 합니다. 나중에 작업한 내용에 대해서는 github 에 올리도록 하겠습니다. 요즘 프로젝트며 운영 업무가 너무 많아서 이것도 겨우 겨우 시간 내서 작업 했내요.
 elasticsearch-analysis-arirang-5.0.1.zip
설치 방법) $ bin/elasticsearch-plugin install --verbose file:///elasticsearch-analysis-arirang/target/elasticsearch-analysis-arirang-5.0.1.zip
설치 로그) -> Downloading file:///elasticsearch-analysis-arirang-5.0.1.zip Retrieving zip from file:///elasticsearch-analysis-arirang-5.0.1.zip [=================================================] 100% - Plugin information: Name: analysis-arirang Description: Arirang plugin Version: 5.0.1 * Classname: org.elasticsearch.plugin.analysis.arirang.AnalysisArirangPlugin -> Installed analysis-arirang
Elasticsearch 실행 로그) $ bin/elasticsearch [2016-11-24T18:49:09,922][INFO ][o.e.n.Node ] [] initializing ... [2016-11-24T18:49:10,083][INFO ][o.e.e.NodeEnvironment ] [aDGu2B9] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [733.1gb], net total_space [930.3gb], spins? [unknown], types [hfs] [2016-11-24T18:49:10,084][INFO ][o.e.e.NodeEnvironment ] [aDGu2B9] heap size [1.9gb], compressed ordinary object pointers [true] [2016-11-24T18:49:10,085][INFO ][o.e.n.Node ] [aDGu2B9] node name [aDGu2B9] derived from node ID; set [node.name] to override [2016-11-24T18:49:10,087][INFO ][o.e.n.Node ] [aDGu2B9] version[5.0.1], pid[56878], build[080bb47/2016-11-11T22:08:49.812Z], OS[Mac OS X/10.12.1/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_72/25.72-b15] [2016-11-24T18:49:11,335][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [aggs-matrix-stats] [2016-11-24T18:49:11,335][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [ingest-common] [2016-11-24T18:49:11,335][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [lang-expression] [2016-11-24T18:49:11,335][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [lang-groovy] [2016-11-24T18:49:11,335][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [lang-mustache] [2016-11-24T18:49:11,336][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [lang-painless] [2016-11-24T18:49:11,336][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [percolator] [2016-11-24T18:49:11,336][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [reindex] [2016-11-24T18:49:11,336][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [transport-netty3] [2016-11-24T18:49:11,336][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded module [transport-netty4] [2016-11-24T18:49:11,336][INFO ][o.e.p.PluginsService ] [aDGu2B9] loaded plugin [analysis-arirang] [2016-11-24T18:49:14,151][INFO ][o.e.n.Node ] [aDGu2B9] initialized [2016-11-24T18:49:14,151][INFO ][o.e.n.Node ] [aDGu2B9] starting ... [2016-11-24T18:49:14,377][INFO ][o.e.t.TransportService ] [aDGu2B9] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300} [2016-11-24T18:49:17,511][INFO ][o.e.c.s.ClusterService ] [aDGu2B9] new_master {aDGu2B9}{aDGu2B9mQ8KkWCe3fnqeMw}{_y9RzyKGSvqYAFcv99HBXg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) [2016-11-24T18:49:17,584][INFO ][o.e.g.GatewayService ] [aDGu2B9] recovered [0] indices into cluster_state [2016-11-24T18:49:17,588][INFO ][o.e.h.HttpServer ] [aDGu2B9] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200} [2016-11-24T18:49:17,588][INFO ][o.e.n.Node ] [aDGu2B9] started
한글형태소분석 실행) $ curl -X POST -H "Cache-Control: no-cache" -H "Postman-Token: 6d392d83-5816-71ad-556b-5cd6f92af634" -d '{ "analyzer" : "arirang_analyzer", "text" : "[한국] 엘라스틱서치 사용자 그룹의 HENRY 입니다." }' "http://localhost:9200/_analyze"
형태소분석 결과) { "tokens": [ { "token": "[", "start_offset": 0, "end_offset": 1, "type": "symbol", "position": 0 }, { "token": "한국", "start_offset": 1, "end_offset": 3, "type": "korean", "position": 1 }, { "token": "]", "start_offset": 3, "end_offset": 4, "type": "symbol", "position": 2 }, { "token": "엘라스틱서치", "start_offset": 5, "end_offset": 11, "type": "korean", "position": 3 }, { "token": "엘라", "start_offset": 5, "end_offset": 7, "type": "korean", "position": 3 }, { "token": "스틱", "start_offset": 7, "end_offset": 9, "type": "korean", "position": 4 }, { "token": "서치", "start_offset": 9, "end_offset": 11, "type": "korean", "position": 5 }, { "token": "사용자", "start_offset": 12, "end_offset": 15, "type": "korean", "position": 6 }, { "token": "그룹", "start_offset": 16, "end_offset": 18, "type": "korean", "position": 7 }, { "token": "henry", "start_offset": 20, "end_offset": 25, "type": "word", "position": 8 }, { "token": "입니다", "start_offset": 26, "end_offset": 29, "type": "korean", "position": 9 } ] }
Elastic/Elasticsearch 2016. 11. 2. 17:49
site plugin head 설치를 해보려 합니다. 설치 방법이 달라졌습니다.
[사전준비] Elasticsearch 5.0 GA 살펴보기 - 1 에서 클러스터 구성 그냥 bin/elasticsearch, bin/elasticsearch 두 번 하시면 됩니다.
[설치 명령어] https://www.elastic.co/guide/en/elasticsearch/plugins/current/installation.html#_core_elasticsearch_plugins
$ bin/elasticsearch-plugin install [plugin-name]
[Head Plugins] https://github.com/mobz/elasticsearch-head#running-with-built-in-server
설치 방법은 위 문서에 잘 나와 있습니다. 아래는 제가 그냥 실행한 명령어를 나열 하였습니다.
$ git clone git://github.com/mobz/elasticsearch-head.git $ cd elasticsearch-head/ $ npm install $ sudo npm install -g grunt-cli $ grunt server
[Head 접속] $ open http://localhost:9100 - cross origin 정책으로 인해서 초기 접속이 되지 않습니다. - elasticsearch.yml 에 아래 내용 추가해 주시고 재실행 하시면 됩니다.
[Cross Origin 정책수정] $ vi config/elasticsearch.yml
http.cors.enabled: true http.cors.allow-origin: "*"
노드명이 많이 아쉽내요 ㅠ.ㅠ
Elastic/Elasticsearch 2016. 11. 2. 17:11
내년에 업그레이드를 해야 하기 때문에 한번 살펴 보고자 합니다.오늘은 가볍게 다운 받고 압축 풀고 기본 구조가 어떻게 변경이 되었나 한번 살펴 보려 합니다.
[다운로드] https://www.elastic.co/downloads/elasticsearch
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.0.0.tar.gz
[압축해제] $ tar -xvzf elasticsearch-5.0.0.tar.gz $ ln -s elasticsearch-5.0.0 elasticsearch
[바로실행] $ bin/elasticsearch
[2016-11-02T16:52:22,051][INFO ][o.e.n.Node ] [] initializing ... [2016-11-02T16:52:22,246][INFO ][o.e.e.NodeEnvironment ] [DiStpli] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [734.4gb], net total_space [930.3gb], spins? [unknown], types [hfs] [2016-11-02T16:52:22,247][INFO ][o.e.e.NodeEnvironment ] [DiStpli] heap size [1.9gb], compressed ordinary object pointers [true] [2016-11-02T16:52:22,248][INFO ][o.e.n.Node ] [DiStpli] node name [DiStpli] derived from node ID; set [node.name] to override [2016-11-02T16:52:22,268][INFO ][o.e.n.Node ] [DiStpli] version[5.0.0], pid[14095], build[253032b/2016-10-26T04:37:51.531Z], OS[Mac OS X/10.12.1/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_72/25.72-b15] [2016-11-02T16:52:23,487][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [aggs-matrix-stats] [2016-11-02T16:52:23,487][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [ingest-common] [2016-11-02T16:52:23,488][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [lang-expression] [2016-11-02T16:52:23,488][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [lang-groovy] [2016-11-02T16:52:23,488][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [lang-mustache] [2016-11-02T16:52:23,488][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [lang-painless] [2016-11-02T16:52:23,488][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [percolator] [2016-11-02T16:52:23,488][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [reindex] [2016-11-02T16:52:23,488][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [transport-netty3] [2016-11-02T16:52:23,488][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [transport-netty4] [2016-11-02T16:52:23,489][INFO ][o.e.p.PluginsService ] [DiStpli] no plugins loaded [2016-11-02T16:52:27,399][INFO ][o.e.n.Node ] [DiStpli] initialized [2016-11-02T16:52:27,399][INFO ][o.e.n.Node ] [DiStpli] starting ... [2016-11-02T16:52:27,809][INFO ][o.e.t.TransportService ] [DiStpli] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300} [2016-11-02T16:52:30,900][INFO ][o.e.c.s.ClusterService ] [DiStpli] new_master {DiStpli}{DiStpli8S4-rE5QKCPP_Lw}{bwHQPV_UTl6NSPCfx8IE3w}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) [2016-11-02T16:52:30,974][INFO ][o.e.h.HttpServer ] [DiStpli] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200} [2016-11-02T16:52:30,974][INFO ][o.e.n.Node ] [DiStpli] started [2016-11-02T16:52:30,975][INFO ][o.e.g.GatewayService ] [DiStpli] recovered [0] indices into cluster_state
[접속]
{
"name" : "DiStpli",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "eA4iUJ4BSoOxmoDbLjFgoQ",
"version" : {
"number" : "5.0.0",
"build_hash" : "253032b",
"build_date" : "2016-10-26T04:37:51.531Z",
"build_snapshot" : false,
"lucene_version" : "6.2.0"
},
"tagline" : "You Know, for Search"
}
[디렉토리구조] $ ls -al total 56 drwxr-xr-x 12 staff 408 11 2 16:52 . drwxr-xr-x 34 staff 1156 11 2 16:45 .. -rw-r--r-- 1 staff 11358 10 26 13:35 LICENSE.txt -rw-r--r-- 1 staff 150 10 26 13:35 NOTICE.txt -rw-r--r-- 1 staff 9108 10 26 13:35 README.textile drwxr-xr-x 15 staff 510 10 26 13:40 bin drwxr-xr-x 6 staff 204 11 2 16:52 config drwxr-xr-x 3 staff 102 11 2 16:52 data drwxr-xr-x 36 staff 1224 10 26 13:40 lib drwxr-xr-x 6 staff 204 11 2 16:52 logs drwxr-xr-x 12 staff 408 10 26 13:40 modules drwxr-xr-x 2 staff 68 11 2 16:52 plugins
- 실행 후 모습 입니다.
[디렉토리 bin] - 실행 파일들이 위치해 있습니다.
[디렉토리 config] - elasticsearch.yml 은 그대로 존재 합니다. - logging.yml 은 log4j2.properties 로 변경 되었습니다. - jvm.options 가 추가 되었습니다. 기존에 실행 시 설정에 필요했던 jvm 옵션을 스크립트에 추가 하기 위해 커스텀하게 사용했는데요. 이 옵션이 생겨서 좀 더 유연해 졌내요. - scripts 디렉토리가 생겼습니다. 잘 아시겠지만 각종 script 관련 코드를 이곳에 저장하고 읽어 들일 수 있도록 된 것 같습니다.
[디렉토리 data] - 예전 그대로 같지만 조금 달라졌습니다. 즉, 예전에는 data/클러스터명/nodes/0 뭐 이런 식이였는데요. 지금은 data/nodes/0 으로 생성이 되내요. 클러스터명이 사라졌내요. (여전히 default cluster_name 은 elasticsearch 입니다.)
[디렉토리 lib] - 바뀐건 종속 라이브러리가 바뀌었내요.
[디렉토리 modules] - elastic 에서 제공하는 추가 modules 들이 들어 있습니다.
[디렉토리 plugins] - 바뀐건 아마도 plugin 만드는 방법이 바뀌었을 것 같습니다.
[초간단 클러스터 구성 테스트] - 기본적으로 elasticsearch 는 bin/elasticsearch 를 여러번 실행 하면 클러스터로 묶어 줍니다. 단, 인스턴스를 분리 하셔야 합니다.
$ bin/elasticsearch [2016-11-02T17:21:04,689][INFO ][o.e.n.Node ] [] initializing ... [2016-11-02T17:21:04,770][INFO ][o.e.e.NodeEnvironment ] [npjHq7P] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [734.4gb], net total_space [930.3gb], spins? [unknown], types [hfs] [2016-11-02T17:21:04,771][INFO ][o.e.e.NodeEnvironment ] [npjHq7P] heap size [1.9gb], compressed ordinary object pointers [true] [2016-11-02T17:21:04,772][INFO ][o.e.n.Node ] [npjHq7P] node name [npjHq7P] derived from node ID; set [node.name] to override [2016-11-02T17:21:04,773][INFO ][o.e.n.Node ] [npjHq7P] version[5.0.0], pid[14680], build[253032b/2016-10-26T04:37:51.531Z], OS[Mac OS X/10.12.1/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_72/25.72-b15] [2016-11-02T17:21:05,375][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [aggs-matrix-stats] [2016-11-02T17:21:05,375][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [ingest-common] [2016-11-02T17:21:05,376][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [lang-expression] [2016-11-02T17:21:05,376][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [lang-groovy] [2016-11-02T17:21:05,376][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [lang-mustache] [2016-11-02T17:21:05,376][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [lang-painless] [2016-11-02T17:21:05,376][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [percolator] [2016-11-02T17:21:05,376][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [reindex] [2016-11-02T17:21:05,376][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [transport-netty3] [2016-11-02T17:21:05,376][INFO ][o.e.p.PluginsService ] [npjHq7P] loaded module [transport-netty4] [2016-11-02T17:21:05,377][INFO ][o.e.p.PluginsService ] [npjHq7P] no plugins loaded [2016-11-02T17:21:06,764][INFO ][o.e.n.Node ] [npjHq7P] initialized [2016-11-02T17:21:06,765][INFO ][o.e.n.Node ] [npjHq7P] starting ... [2016-11-02T17:21:06,895][INFO ][o.e.t.TransportService ] [npjHq7P] publish_address {127.0.0.1:9301}, bound_addresses {[fe80::1]:9301}, {[::1]:9301}, {127.0.0.1:9301} [2016-11-02T17:21:09,996][INFO ][o.e.c.s.ClusterService ] [npjHq7P] detected_master {DiStpli}{DiStpli8S4-rE5QKCPP_Lw}{oJ9y-SxJTYex3ZNBmRIpoQ}{127.0.0.1}{127.0.0.1:9300}, added {{DiStpli}{DiStpli8S4-rE5QKCPP_Lw}{oJ9y-SxJTYex3ZNBmRIpoQ}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [master {DiStpli}{DiStpli8S4-rE5QKCPP_Lw}{oJ9y-SxJTYex3ZNBmRIpoQ}{127.0.0.1}{127.0.0.1:9300} committed version [7]]) [2016-11-02T17:21:10,030][INFO ][o.e.h.HttpServer ] [npjHq7P] publish_address {127.0.0.1:9201}, bound_addresses {[fe80::1]:9201}, {[::1]:9201}, {127.0.0.1:9201} [2016-11-02T17:21:10,031][INFO ][o.e.n.Node ] [npjHq7P] started
$ bin/elasticsearch [2016-11-02T17:19:34,869][INFO ][o.e.n.Node ] [] initializing ... [2016-11-02T17:19:35,074][INFO ][o.e.e.NodeEnvironment ] [DiStpli] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [734.4gb], net total_space [930.3gb], spins? [unknown], types [hfs] [2016-11-02T17:19:35,074][INFO ][o.e.e.NodeEnvironment ] [DiStpli] heap size [1.9gb], compressed ordinary object pointers [true] [2016-11-02T17:19:35,076][INFO ][o.e.n.Node ] [DiStpli] node name [DiStpli] derived from node ID; set [node.name] to override [2016-11-02T17:19:35,079][INFO ][o.e.n.Node ] [DiStpli] version[5.0.0], pid[14588], build[253032b/2016-10-26T04:37:51.531Z], OS[Mac OS X/10.12.1/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_72/25.72-b15] [2016-11-02T17:19:37,110][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [aggs-matrix-stats] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [ingest-common] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [lang-expression] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [lang-groovy] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [lang-mustache] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [lang-painless] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [percolator] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [reindex] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [transport-netty3] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] loaded module [transport-netty4] [2016-11-02T17:19:37,111][INFO ][o.e.p.PluginsService ] [DiStpli] no plugins loaded [2016-11-02T17:19:40,144][INFO ][o.e.n.Node ] [DiStpli] initialized [2016-11-02T17:19:40,145][INFO ][o.e.n.Node ] [DiStpli] starting ... [2016-11-02T17:19:40,356][INFO ][o.e.t.TransportService ] [DiStpli] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2016-11-02T17:19:43,625][INFO ][o.e.c.s.ClusterService ] [DiStpli] new_master {DiStpli}{DiStpli8S4-rE5QKCPP_Lw}{oJ9y-SxJTYex3ZNBmRIpoQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) [2016-11-02T17:19:43,742][INFO ][o.e.g.GatewayService ] [DiStpli] recovered [0] indices into cluster_state [2016-11-02T17:19:43,750][INFO ][o.e.h.HttpServer ] [DiStpli] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200} [2016-11-02T17:19:43,750][INFO ][o.e.n.Node ] [DiStpli] started [2016-11-02T17:21:09,966][INFO ][o.e.c.s.ClusterService ] [DiStpli] added {{npjHq7P}{npjHq7PlTqqiB2bhXkIxqQ}{IhQPGv_rTsyd7a1DbXFMwA}{127.0.0.1}{127.0.0.1:9301},}, reason: zen-disco-node-join[{npjHq7P}{npjHq7PlTqqiB2bhXkIxqQ}{IhQPGv_rTsyd7a1DbXFMwA}{127.0.0.1}{127.0.0.1:9301}] [2016-11-02T17:21:10,007][WARN ][o.e.d.z.ElectMasterService] [DiStpli] value for setting "discovery.zen.minimum_master_nodes" is too low. This can result in data loss! Please set it to at least a quorum of master-eligible nodes (current value: [-1], total number of master-eligible nodes used for publishing in this round: [2]) - 친절하게 quorum 구성 하라고 메시지도 보여 주내요.
Elastic/Elasticsearch 2016. 11. 1. 13:33
요즘 추천 데이터 만드느라 도통 mapping 정보 설계나 dsl 작성을 안했더니 까먹기 일수내요. 그래서 그냥 기억하는 차원에서 가볍게 작성해 봅니다.
[참고문서] https://www.elastic.co/guide/en/elasticsearch/reference/2.4/multi-fields.html
[설명] - 예전에는 multi-field 라고 했고 지금은 fields 라고 합니다. - fields 의 용도는 단일 field 에 여러가지 속성을 부여하고 싶을 때 사용을 합니다.
예를 들면) - index:analyzed 로 선언을 했는데 정렬을 해야 할 경우. - index:not_analyzed 로 선언을 했는데 같은 value로 fulltext 검색을 해야 할 경우.
보통은 index:analyzed 로 선언 하고 fields 로 index:not_analyzed 로 구성 하는게 많습니다.
[mappings 정보] ...중략... "keyword": { "type":"string", "index":"analyzed", "analyzer":"edge_ngram_analyzer", "index_options":"docs", "norms": { "enabled":false }, "fields": { "exact": { "type":"string", "index":"not_analyzed", "store":"no" } } }, ...중략... - 위 맵핑 정보는 제가 자동완성에서 사용하는 정보를 일부 발췌한 내용입니다.
[query dsl] { "query": { "term": { "keyword.exact": "향수" } } } - 질의는 위에서와 같이 .(dot) 을 이용해서 field 명을 사용하시면 됩니다.
Elastic/Elasticsearch 2016. 8. 10. 16:42
trade-off 가 있긴 하지만 그럼에도 불구하고 색인 성능을 확보 하고 싶으신 분들은 한번 고려해 보셔도 될 것 같습니다.
기본 설정은 sync 입니다.
index.translog.durability: async
추가적으로 software 적인 raid 구조를 사용하고 계시다면, disk i/o 성능이 떨어 지게 되기 때문에 사용하지 않는 걸 추천 드립니다.
|
