jjeong

https://www.elastic.co/guide/en/elasticsearch/guide/current/_exact_value_fields.html

Avoid using not_analyzed fields in multi_match queries.

However, for performance reasons, Elasticsearch doesn’t calculate the IDF across all documents in the index. Instead, each shard calculates a local IDF for the documents contained in that shard.

...중략...

Don’t use dfs_query_then_fetch in production. It really isn’t required. Just having enough data will ensure that your term frequencies are well distributed. There is no reason to add this extra DFS step to every query that you run.

1. Check the field type

2. Analyze the query string (term query 로 재실행 됩니다.)

3. Find matching docs

4. Score each doc

1. Find matching docs

2. Score each doc

"range" : { "timestamp" : { "gt" : "2014-01-01 00:00:00", "lt" : "2014-01-07 00:00:00" } }

"range" : { "timestamp" : { "gt" : "now-1h" } }

"range" : { "timestamp" : { "gt" : "2014-01-01 00:00:00", "lt" : "2014-01-01 00:00:00||+1M" } }

With the automatic refresh process creating a new segment every second, it doesn’t take long for the number of segments to explode.

...중략...

This is the moment when those old deleted documents are purged from the filesystem. Deleted documents (or old versions of updated documents) are not copied over to the new bigger segment.

...중략...

The merge process... does not interrupt indexing and searching.

...중략...

Once merging has finished, the old segments are deleted

1) merge 하기 위한 new segment 를 생성 합니다. (run optimize)

2) deleted mark 된 것들을 제외 하고 신규 segment 로 merge 대상 segment 들이 합쳐 집니다.)

3) 신규 commit point 를 생성 합니다. (merge 대상 segment 는 제거 하고, 신규 segment 와 아직 merge 가 안된 segment 정보만 기록 합니다.)

4) 검색 가능한 상태가 됩니다.

5) merge 가 완료된 segment는 삭제 됩니다.

Elasticsearch uses this commit point during startup or when reopening an index to decide which segments belong to the current shard.

...중략...

Elasticsearch added a translog, or transaction log, which records every operation in Elasticsearch as it happens.

1) write in-memory buffer & translog

2) write new segment file without fsync

3) flush in-memory buffer (run refresh and not yet flush)

4) write in-memory buffer & append translog 

5) write new segment file (run flush)

6) flush in-memory buffer

7) write commit point on disk

8) flush filesystem cache with fsync

9) delete old translog

10) create new translog

검색 가능한 상태로 만들어 줍니다.

fsync 작업을 합니다. (commit point 기록 및 translog 제거)

[Deletes and Updates]

Segments are immutable, so documents cannot be removed from older segments, nor can older segments be updated to reflect a newer version of a document. Instead, every commit point includes a .del file that lists which documents in which segments have been deleted.

When a document is “deleted,” it is actually just marked as deleted in the .del file. A document that has been marked as deleted can still match a query, but it is removed from the results list before the final query results are returned.

Document updates work in a similar way: when a document is updated, the old version of the document is marked as deleted, and the new version of the document is indexed in a new segment. Perhaps both versions of the document will match a query, but the older deleted version is removed before the query results are returned.

1) write in-memory buffer

2) write new segment file from in-memory buffer

3) write commit point (Refresh API on Elasticsearch)

4) flush in-memory buffer

5) searchable

PUT /my_index { "mappings": { "my_type": { "date_detection": false } } }

https://www.elastic.co/guide/en/elasticsearch/guide/current/_index_settings.html

Elasticsearch comes with good defaults. Don’t twiddle these knobs until you understand what they do and why you should change them.