jjeong

그냥 지나가다가...

질의 성능 최적화에서 queries vs filters 에 대해 이해하고 사용하고 계실줄로 압니다만, ^^;

요기에 추가로 bitset 에 대해서도 내용이 나오기 때문에 그냥 참고하시라고 링크 던져 봅니다. ^^

- segment merge policy

- 검색 질의에 대한 유형 및 데이터 크기

- field, filter 관련 cache 관리

- facet, aggregation에 대한 유형 및 데이터 크기

- jdk 7 이상 사용

- 등등등..

-server

-XX:+AggressiveOpts

-XX:UseCompressedOops

-XX:MaxDirectMemorySize

-XX:+UseParNewGC

-XX:+UseConcMarkSweepGC 

-XX:+CMSParallelRemarkEnabled

-XX:CMSInitiatingOccupancyFraction=75

-XX:+UseCMSInitiatingOccupancyOnly

-XX:+UseG1GC

ZenDiscovery

        ↓

MasterFaultDetection

        ↓

ElectMasterService

public class EsHashPartitionTest {

    private static final Logger log = LoggerFactory.getLogger(EsHashPartitionTest.class);

    private HashFunction hashFunction = new DjbHashFunction();

    @Test

    public void testHashPartition() {

        int shardSize = 120;

        List<Long> shards = new ArrayList<Long>();

        long[] partSize = new long[shardSize];

        for ( int i=0; i<shardSize; i++ ) {

            shards.add((long) 0);

            partSize[i] = 0;

        }

        for ( int i=0; i<1000000; i++ ) {

            int shardId = MathUtils.mod(hash(String.valueOf(i)), shardSize);

            shards.add(shardId, (long) ++partSize[shardId]);

        }

        for ( int i=0; i<shardSize; i++ ) {

            log.debug("["+i+"] {}", partSize[i]);

        }

    }

    public int hash(String routing) {

        return hashFunction.hash(routing);

    }

}

/**

 * This class implements the efficient hash function

 * developed by <i>Daniel J. Bernstein</i>.

 */

public class DjbHashFunction implements HashFunction {

    public static int DJB_HASH(String value) {

        long hash = 5381;

        for (int i = 0; i < value.length(); i++) {

            hash = ((hash << 5) + hash) + value.charAt(i);

        }

        return (int) hash;

    }

    public static int DJB_HASH(byte[] value, int offset, int length) {

        long hash = 5381;

        final int end = offset + length;

        for (int i = offset; i < end; i++) {

            hash = ((hash << 5) + hash) + value[i];

        }

        return (int) hash;

    }

    @Override

    public int hash(String routing) {

        return DJB_HASH(routing);

    }

    @Override

    public int hash(String type, String id) {

        long hash = 5381;

        for (int i = 0; i < type.length(); i++) {

            hash = ((hash << 5) + hash) + type.charAt(i);

        }

        for (int i = 0; i < id.length(); i++) {

            hash = ((hash << 5) + hash) + id.charAt(i);

        }

        return (int) hash;

    }

}

Controls a preference of which shard replicas to execute the search request on. By default, the operation is randomized between the shard replicas.

The preference is a query string parameter which can be set to:

Elasticsearch

      |

      V

Bootstrap

      |

      V

NodeBuilder

      |

      V

InternalNode

The merging of big segments can use a lot of I/O and CPU, which can hurt search performance if left unchecked. By default, Elasticsearch throttles the merge process so that search still has enough resources available to perform well.

Be aware that merges triggered by the optimize API are not throttled at all. They can consume all of the I/O on your nodes, leaving nothing for search and potentially making your cluster unresponsive. If you plan on optimizing an index, you should use shard allocation (see Appendix A, TODO) to first move the index to a node where it is safe to run.

The entire bulk request needs to be loaded into memory by the node which receives our request, so the bigger the request, the less memory available for other requests. There is an optimal size of bulk request. Above that size, performance no longer improves and may even drop off.

The optimal size, however, is not a fixed number. It depends entirely on your hardware, your document size and complexity, and your indexing and search load. Fortunately, it is easy to find this sweetspot:

Try indexing typical documents in batches of increasing size. When performance starts to drop off, your batch size is too big. A good place to start is with batches of between 1,000 and 5,000 documents or, if your documents are very large, with even smaller batches.

It is often useful to keep an eye on the physical size of your bulk requests. One thousand 1kB documents is very different than one thousand 1MB documents. A good bulk size to start playing with is around 5-15MB in size.

In Elasticsearch, this lightweight process of writing and opening a new segment is called a refresh. By default, every shard is refreshed automatically once every second. This is why we say that Elasticsearch has near real-time search: document changes are not visible to search immediately, but will become visible within one second.

The action of performing a commit and truncating the translog is known in Elasticsearch as a flush. Shards are flushed automatically every 30 minutes, or when the translog becomes too big. See thetranslog documentation for settings that can be used to control these thresholds.

The optimize API is best described as the forced merge API. It forces a shard to be merged down to the number of segments specified in the max_num_segments parameter. The intention is to reduce the number of segments (usually to 1) in order to speed up search performance.