'Elastic/Elasticsearch'에 해당되는 글 385건

  1. 2013.04.05 [elasticsearch] Java API : Client
  2. 2013.04.03 [elasticsearch] plugin elasticsearch-hadoop 설치하기.
  3. 2013.04.03 [elasticsearch] tip, 초간단 es start/stop script.
  4. 2013.04.03 [elasticsearch] tip, 0.17.10 on macosx - initialization failed
  5. 2013.02.25 elasticsearch book published!!
  6. 2013.02.18 split brain to elasticsearch.
  7. 2013.02.13 This is about elasticsearch optimization.
  8. 2013.02.08 elasticsearch multi field type
  9. 2013.02.05 elasticsearch-jetty plugin run tips
  10. 2013.02.02 Elasticsearch jetty plugin + authenticate

[elasticsearch] Java API : Client

Elastic/Elasticsearch 2013. 4. 5. 19:01

본 문서는 개인적인 테스트와 elasticsearch.org 그리고 community 등을 참고해서 작성된 것이며,

정보 교환이 목적입니다.


잘못된 부분에 대해서는 지적 부탁 드립니다.

(예시 코드는 성능 및 보안 검증이 되지 않았습니다.)



[elasticsearch java api 리뷰]

원문 링크 : http://www.elasticsearch.org/guide/reference/java-api/client/


Java API 를 이용해서 elasticsearch 로 연결하는 두 가지 방법을 설명 하고 있습니다.

1. Node Client

2. Transport Client


※ 커뮤니티에서 관련 글들을 찾다 보면 kimchy 가 추천 하는 방식은 "TransportClient" 를 이용하는 것입니다.


원문에서 보면 elasticsearch 를 사용하면 주의 해야 하는 부분이 나와 있습니다.

사용하시면서 꼭 숙지 하시면 한번 쯤 실수 할 수 있는 오류를 범하지 않을 수 있습니다.

Important:

Please note that you are encouraged to use the same version on client and cluster sides. You may hit some incompatibilities issues when mixing major versions.



※ 요점은 버전을 혼합해서 사용하지 말라는 것입니다. (No Node Available Exception 의 이유가 됩니다.)


원문에 예제 코드가 잘 나와 있기 때문에 제가 테스트 한 코드를 아래 공유 합니다.

ClientConfig clientConfig = ClientConfig.getInstance();


// cluster.name 을 안 줄 경우 NoNodeAvailable 에러 발생.

settings = ImmutableSettings

        .settingsBuilder()

        .put("cluster.name", clientConfig.getProperty("cluster.name"))

        .build();

client = new TransportClient(settings)

        .addTransportAddress(new InetSocketTransportAddress(clientConfig.getProperty("cluster.host1")

                , Integer.parseInt(clientConfig.getProperty("cluster.host1.port"))))

        .addTransportAddress(new InetSocketTransportAddress(clientConfig.getProperty("cluster.host2")

                , Integer.parseInt(clientConfig.getProperty("cluster.host2.port"))));


- ClientConfig : elasticsearch 를 사용하기 위해 필요한 환경설정 값을 구성


elasticsearch client 가 가지는 properties 와 attributes 테스트를 위한 코드

@Test

public void testClientProperties() throws Exception {

    clientManager.openClient();

    Client client = clientManager.getClient();

    AdminClient admin = client.admin();

    ClusterAdminClient cluster = admin.cluster();

    IndicesAdminClient indices = admin.indices();


    ClusterStateRequestBuilder request = cluster.prepareState();

    ClusterStateResponse response = request.execute().actionGet();  

    ClusterState state = response.state();

    AllocationExplanation allocation = state.allocationExplanation();

    ClusterBlocks block = state.blocks();

    MetaData meta = state.metaData();

    RoutingNodes routingNode = state.routingNodes();

    RoutingTable routingTable = state.routingTable();

    ImmutableSettings settings = (ImmutableSettings) meta.persistentSettings();

    

    log.debug("{}", response.clusterName()); // Cluster [test.gruter.es.cluster]

    log.debug("{}", response.getClusterName());

    log.debug("{}", response.getHeaders());

    log.debug("{}", response.getState());

    log.debug("{}", response.state());

    

    log.debug("{}", state.allocationExplanation());

    log.debug("{}", state.blocks());

    log.debug("{}", state.metaData());

    log.debug("{}", state.nodes()); // {[test.gruter.es.cluster2.node][sIj29f-KQziaIbccn_jndg][inet[/127.0.0.1:9301]]{master=true},[test.gruter.es.cluster1.node][kWNDF8SLTxS_axatGmImIg][inet[/127.0.0.1:9300]]{master=true},}

    log.debug("{}", state.routingNodes());

    log.debug("{}", state.routingTable());

    log.debug("{}", state.version());

    

    log.debug("{}", allocation.explanations());

    

    log.debug("{}", block.indices());

    

    log.debug("{}", meta.concreteAllIndices());

    log.debug("{}", meta.concreteAllOpenIndices());

    log.debug("{}", meta.getTotalNumberOfShards());

    log.debug("{}", meta.persistentSettings());

    log.debug("{}", meta.settings());

    

    log.debug("{}", routingNode.getMetaData());

    log.debug("{}", routingNode.getNodesToShards());

    log.debug("{}", routingNode.getRoutingTable());

    log.debug("{}", routingNode.getUnassigned());

    log.debug("{}", routingNode.prettyPrint());

    

    log.debug("{}", routingTable.getIndicesRouting());

    log.debug("{}", routingTable.prettyPrint());

    

    log.debug("{}", settings.getAsMap());

    log.debug("{}", state.nodes().dataNodes());

    log.debug("{}", state.nodes().getSize());

   

    for ( int i=0; i<state.nodes().getSize(); i++ ) {

        log.debug("{}", state.nodes().dataNodes().keySet().toArray()[i]);

    }

    

    Set entries = state.nodes().dataNodes().entrySet();

    Iterator entryIter = entries.iterator();


    while (entryIter.hasNext()) {

        Map.Entry entry = (Map.Entry)entryIter.next();

        Object key = entry.getKey();

        Object value = entry.getValue();

       

        log.debug("key : {}", key);

        log.debug("value : {}", value);

    }

} 

※ 단순히 테스트 및 각 속성들을 보기 위한 코드 입니다.

:

[elasticsearch] plugin elasticsearch-hadoop 설치하기.

Elastic/Elasticsearch 2013. 4. 3. 17:13

Main Site : http://www.elasticsearch.org/guide/reference/modules/gateway/hadoop/

Plugin Site : https://github.com/elasticsearch/elasticsearch-hadoop



bin/plugin -install elasticsearch/elasticsearch-hadoop/1.2.0



:

[elasticsearch] tip, 초간단 es start/stop script.

Elastic/Elasticsearch 2013. 4. 3. 11:37

매번 여러가지 옵션을 주고 실행 시키고, 프로세스 확인해서 죽이고 하기 번거롭죠.

그래서 그냥 간편하게 활용해 보시라고 올려 봅니다.

es 0.17.10 으로 테스트 한거고 아래 스크립트로 0.20.x 나 0.19.x 다 됩니다.


jeong-ui-MacBook-Pro:bin hwjeong$ cat start_es.sh 

#!/bin/bash


ES_PATH=/Users/hwjeong/server/app/es-0.17.10


$ES_PATH/bin/elasticsearch -p $ES_PATH/bin/pidfile.pid -Des.config=$ES_PATH/config/elasticsearch.yml -Xmx256m -Xms256m -Des.max-open-files=true > /dev/null 2>&1 &

jeong-ui-MacBook-Pro:bin hwjeong$ cat stop_es.sh 

#!/bin/bash


ES_PATH=/Users/hwjeong/server/app/es-0.17.10


/bin/kill `cat < $ES_PATH/bin/pidfile.pid `


:

[elasticsearch] tip, 0.17.10 on macosx - initialization failed

Elastic/Elasticsearch 2013. 4. 3. 11:19


적용된 es 가 0.17.10 이라서 mac 에서 압축 풀고 실행했더니 헐.. 바로 오류 던지싶니다.


[2013-04-03 10:56:11,568][INFO ][node                     ] [Halloway, Thomas] {elasticsearch/0.17.10}[638]: initializing ...

[2013-04-03 10:56:11,576][INFO ][plugins                  ] [Halloway, Thomas] loaded [], sites []

[2013-04-03 10:56:14,456][ERROR][bootstrap                ] {elasticsearch/0.17.10}: Initialization Failed ...


kimchy 의 recommendation 은 버전을 올리거나 elasticsearch.in.sh 에서 아래 line 을 수정 하라는 내용입니다.

뭐 이제 이 버전을 사용하고 계신 분들은 없겠지만 그냥 올려 봅니다.


# http://stackoverflow.com/questions/12403489/elasticsearch-os-x-could-not-initialize-class-org-elasticsearch-common-xcontent


this line : JAVA_OPTS="$JAVA_OPTS -Xss128k"

to : JAVA_OPTS="$JAVA_OPTS -Xss256k"


:

elasticsearch book published!!

Elastic/Elasticsearch 2013. 2. 25. 18:21
:

split brain to elasticsearch.

Elastic/Elasticsearch 2013. 2. 18. 18:04

Prevent a split-brain to elasticsearch.


[Recommended Configuration]

[node1]

minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts: [node2, node1]

 

[node2]

minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts: [node1, node2]


split-brain situations?

In case of Clustering

- problem network communications.

- corrupt data of each nodes.


Other references.

http://en.wikipedia.org/wiki/High-availability_cluster

HA clusters usually use a heartbeat private network connection which is used to monitor the health and status of each node in the cluster. One subtle but serious condition all clustering software must be able to handle is split-brain, which occurs when all of the private links go down simultaneously, but the cluster nodes are still running. If that happens, each node in the cluster may mistakenly decide that every other node has gone down and attempt to start services that other nodes are still running. Having duplicate instances of services may cause data corruption on the shared storage.


:

This is about elasticsearch optimization.

Elastic/Elasticsearch 2013. 2. 13. 16:11

Summary is,

- JVM heap size is setup 50% on your system memory.

- index store type is setup by mmapfs. (in case of 64bits solaris)

- adjust index cache config. 

- adjust index merge config.

 

http://blog.bugsense.com/post/35580279634/indexing-bigdata-with-elasticsearch


http://www.slideshare.net/kucrafal/scaling-massive-elastic-search-clusters-rafa-ku-sematext

Page 27.

    index.cache.field.max_size

    index.cache.field.expire

    index.cache.field.type: soft

Page 30.

    JVM Optimization

    -XX:+UseParNewGC

    -XX:+UseConcMarkSweepGC

    -XX:+CMSParallelRemarkEnabled

    

https://gist.github.com/deverton/2970285


JVM Swap 방지

    bootstrap.mlockall: true


http://stackoverflow.com/questions/13757398/elasticsearch-bulk-indexing-gets-slower-over-time-with-constant-number-of-indexe    

    Set the ES_HEAP_SIZE environment variable so that the JVM uses the same value for minimum and maximum memory. Configuring the JVM to have different minimum and maximum values means that each time the JVM needs additional memory (up to the maximum), it will block the Java process to allocate it. Combined with the old Java version, this explains the pauses that our nodes exhibited when introduced to higher load and continuous memory allocation when they were opened up to public searches. The elasticsearch team recommends a setting of 50% of system RAM.    

    

http://www.elasticsearch.org/guide/reference/index-modules/store.html

        using  index.store.type: mmapfs

        

http://jprante.github.com/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html  

 

       





:

elasticsearch multi field type

Elastic/Elasticsearch 2013. 2. 8. 12:04

http://www.elasticsearch.org/guide/reference/mapping/multi-field-type.html


It needs a sorting field.


"subject": { "type": "multi_field", "fields": { "subject": { "type": "string", "index": "analyzed", "store": "yes", "analyzer": "kr_analyzer" }, "sort_subject": { "include_in_all": false, "type": "string", "index": "not_analyzed", "store": "yes" } } },


:

elasticsearch-jetty plugin run tips

Elastic/Elasticsearch 2013. 2. 5. 17:13

I have installed the elasticsearch-jetty plugin on my box.

My box is configured by single instance using elasticsearch.

By the way, It's not working on my box.

hmm.... 


So, I analyzed the configuration that elasticsearch-jetty is working.

Finally, I found difference assets.


[Not Working Configure]

- It just configured used Single Instance.


[Working Configure] 

- It just configured used Multiple Instance.


I don't know what is the exact reason.

I'll deep dive the problem soon.


I referenced the below url.

https://github.com/sonian/elasticsearch-jetty

:

Elasticsearch jetty plugin + authenticate

Elastic/Elasticsearch 2013. 2. 2. 22:38

I didn't install this plugin(elasticsearch-jetty) but it will apply on my ES soon.
More detail information read the below url.

[Reference]
https://github.com/sonian/elasticsearch-jetty

[Authenticate]
realm.properties

: