'kibana'에 해당되는 글 33건

  1. 2020.08.26 [Elastic] Elastic Agent 테스트.
  2. 2020.06.19 [Elasticsearch] X-pack Security API Key 사용 해 보기
  3. 2020.06.17 [Elasticsearch] script 사용 시 "#! Deprecation: Deprecated field [inline] used, expected [source] instead"
  4. 2020.05.11 [Kibana] Docker Compose 실행 시 Permission 오류.
  5. 2020.04.07 [Kibana] Visualize Metric JSON Input 예제
  6. 2020.04.02 [Kibana] Docker Compose 구성 하기
  7. 2020.04.01 [Kibana] Docker Compose 설정 시 environment 중
  8. 2019.11.05 [Elastic] 목차 입니다.
  9. 2019.04.25 [Kibana] visualize 에서 scripted field 사용하기
  10. 2018.07.05 [Kibana] start and stop 스크립트

[Elastic] Elastic Agent 테스트.

Elastic 2020. 8. 26. 15:10

[참고문서]

https://www.elastic.co/downloads/elastic-agent

https://www.elastic.co/guide/en/ingest-management/current/index.html

 

Elastic Agent 를 사용해 보려고 합니다.

최근에 릴리즈 된 Elastic Stack 7.9.0 으로 진행 했습니다.

 

진행 간에 순서대로 실행한 내용을 그냥 올려 둡니다.

$ bin/elasticsearch -d -p pid
$ bin/elasticsearch-setup-passwords interactive
에러 발생)
{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node."
      }
    ],
    "type" : "exception",
    "reason" : "Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node."
  },
  "status" : 500
}

xpack security 설정)
$ vi config/elasticsearch.yml
xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true

재실행)
$ bin/elasticsearch-setup-passwords interactive
$ bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N] y


Enter password for [elastic]: elastic
Reenter password for [elastic]: elastic
Enter password for [apm_system]: elastic
Reenter password for [apm_system]: elastic
Enter password for [kibana_system]: elastic
Reenter password for [kibana_system]: elastic
Enter password for [logstash_system]: elastic
Reenter password for [logstash_system]: elastic
Enter password for [beats_system]: elastic
Reenter password for [beats_system]: elastic
Enter password for [remote_monitoring_user]: elastic
Reenter password for [remote_monitoring_user]: elastic
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

Kibana 설정)
$ vi config/kibana.yml
46 elasticsearch.username: "kibana_system"
47 elasticsearch.password: "elastic"

$ bin/kibana

Kibana 접속)
http://localhost:5601
  ID/PWD
  elastic/elastic

Kibana IngestManager 접속)
http://localhost:5601/app/ingestManager#/

Agent 사용을 위한 Kibana 추가 설정)
xpack.ingestManager.fleet.tlsCheckDisabled: true
xpack.encryptedSavedObjects.encryptionKey: a123456789012345678901234567890b

이후 단계는 아래 문서 대로 따라 합니다.)
https://www.elastic.co/guide/en/ingest-management/current/ingest-management-getting-started.html
./elastic-agent enroll http://localhost:5601 cEpibEtIUUI2akhoRVR0LWdwZVg6blhHNGdMeURUbVNPQUE0RS1GOFduUQ==
./elastic-agent run

$  ./elastic-agent enroll http://localhost:5601 cEpibEtIUUI2akhoRVR0LWdwZVg6blhHNGdMeURUbVNPQUE0RS1GOFduUQ==
The Elastic Agent is currently in BETA and should not be used in production
This will replace your current settings. Do you want to continue? [Y/n]: y
Error: connection to Kibana is insecure, strongly recommended to use a secure connection (override with --insecure)

$ ./elastic-agent enroll http://localhost:5601 cEpibEtIUUI2akhoRVR0LWdwZVg6blhHNGdMeURUbVNPQUE0RS1GOFduUQ== --insecure
The Elastic Agent is currently in BETA and should not be used in production
This will replace your current settings. Do you want to continue? [Y/n]:y
2020-08-26T14:06:01.782+0900  DEBUG kibana/client.go:170  Request method: POST, path: /api/ingest_manager/fleet/agents/enroll
Successfully enrolled the Agent.

$ ./elastic-agent run

Kibana IngestManager 확인)
http://localhost:5601/app/ingestManager#/

Elasticsearch + Kibana + ElasticAgent + Elastic Common Schema 구성이며, 기존에 beats 를 이용해서 log/data shipper 기능을 구현 하던걸 agent 하나로 대체 할 수 있을 것 같습니다.

더불어서 ECS 기반의 Kibana 에서 Dashboard 와 Visualize 에 대해서도 Preset 을 제공 하니 잘 활용하면 아주 멋질 것으로 기대 합니다.

:

[Elasticsearch] X-pack Security API Key 사용 해 보기

Elastic/Elasticsearch 2020. 6. 19. 11:07

Elastic Stack 이 좋은 이유는 기본 Basic license 까지 사용이 가능 하다는 것입니다.

사실 이것 말고도 엄청 많죠 ㅎㅎ 

 

https://www.elastic.co/subscriptions

 

딱 API keys management 까지 사용이 됩니다. ㅎㅎㅎ

 

먼저 사용하기에 앞서서 Elasticsearch 와 Kibana 에 x-pack 사용을 위한 설정을 하셔야 합니다.

 

[Elasticsearch]

- elasticsearch.yml

xpack.monitoring.enabled: true
xpack.ml.enabled: true
xpack.security.enabled: true

xpack.security.authc.api_key.enabled: true
xpack.security.authc.api_key.hashing.algorithm: "pbkdf2"
xpack.security.authc.api_key.cache.ttl: "1d"
xpack.security.authc.api_key.cache.max_keys: 10000
xpack.security.authc.api_key.cache.hash_algo: "ssha256"

위 설정은 기본이기 때문에 환경에 맞게 최적화 하셔야 합니다.

https://www.elastic.co/guide/en/elasticsearch/reference/7.8/security-settings.html#api-key-service-settings

 

[Kibana]

- kibana.yml

xpack:
  security:
    enabled: true
    encryptionKey: "9c42bff2e04f9b937966bda03e6b5828"
    session:
      idleTimeout: 600000
    audit:
      enabled: true

 

이렇게 설정 한 후 id/password 설정을 하시면 됩니다.

 

# bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]

Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

 

이렇게 설정이 끝나면 kibana 에 접속해서 API key 를 생성 하시면 됩니다.

아래 문서는 생성 시 도움이 되는 문서 입니다.

 

www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html

www.elastic.co/guide/en/elasticsearch/reference/7.7/security-api-put-role.htmlwww.elastic.co/guide/en/elasticsearch/reference/7.7/defining-roles.htmlwww.elastic.co/guide/en/elasticsearch/reference/7.7/security-api-create-api-key.html

 

Kibana Console 에서 아래와 같이 생성이 가능 합니다.

POST /_security/api_key
{
  "name": "team-index-command",
  "expiration": "10m", 
  "role_descriptors": { 
    "role-team-index-command": {
      "cluster": ["all"],
      "index": [
        {
          "names": ["*"],
          "privileges": ["all"]
        }
      ]
    }
  }
}

{
  "id" : "87cuynIBjKAXtnkobGgo",
  "name" : "team-index-command",
  "expiration" : 1592529999478,
  "api_key" : "OlVGT_Q8RGq1C_ASHW7pGg"
}

생성 이후 사용을 위해서는 

 

- ApiKey 는 id:api_key 를 base64 인코딩 합니다.

base64_encode("87cuynIBjKAXtnkobGgo"+":"+"OlVGT_Q8RGq1C_ASHW7pGg")
==> VGVVOXluSUJHUUdMaHpvcUxDVWo6aUtfSmlEMmdSMy1FUUFpdENCYzF1QQ==
curl -H 
  "Authorization: ApiKey VGVVOXluSUJHUUdMaHpvcUxDVWo6aUtfSmlEMmdSMy1FUUFpdENCYzF1QQ==" 
  http://localhost:9200/_cluster/health

이제 용도와 목적에 맞춰서 API key 를 만들고 사용 하시면 되겠습니다.

 

:

[Elasticsearch] script 사용 시 "#! Deprecation: Deprecated field [inline] used, expected [source] instead"

Elastic/Elasticsearch 2020. 6. 17. 08:20

에러 메시지를 보면 답이 나와 있습니다.

inline 대신 source 를 사용 하라는 이야기 입니다.

 

[ASIS]

  "aggs": {
    "3": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30s",
        "time_zone": "Asia/Seoul",
        "min_doc_count": 1
      },
      "aggs": {
        "1": {
          "max": {
            "field": "system.cpu.total.pct",
            "script": {
              "inline": "doc['system.cpu.total.pct'].value *100",
              "lang": "painless"
            }
          }
        }
      }
    }
  }

 

[TOBE]

  "aggs": {
    "3": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30s",
        "time_zone": "Asia/Seoul",
        "min_doc_count": 1
      },
      "aggs": {
        "1": {
          "max": {
            "field": "system.cpu.total.pct",
            "script": {
              "source": "doc['system.cpu.total.pct'].value *100",
              "lang": "painless"
            }
          }
        }
      }
    }
  }

이상 끝.

:

[Kibana] Docker Compose 실행 시 Permission 오류.

Elastic/Kibana 2020. 5. 11. 14:15

kibana 를 docker-compose 를 이용해서 실행 할 때 설정 파일, kibana.yml 파일에 대한 permission 오류가 발생 할 때가 있습니다.

이 경우 처음 부터 kibana.yml 파일이 있는 위치에 permission 을 부여 해 주거나 docker-compose.yml 파일에서 아래와 같이 추가 하시면 문제를 해결 할 수 있습니다.

 

user: "1002:0"

 

- host 서버에서

$ id

uid=1002 gid=20

 

- container 에서 

$ id

uid=1000 gid=0

 

실행한 놈과 실행 된 놈의 user 가 달라서 발생 하는 오류 입니다.

권한을 주거나 user 를 맞춰 주시면 됩니다.

:

[Kibana] Visualize Metric JSON Input 예제

Elastic/Kibana 2020. 4. 7. 20:11

kibana 에서 visualize 중 metric 을 구성 할 때 또는 다른 visualize 이더라도 비슷 합니다.

JSON Input 은 아래와 같이 넣으 실 수 있습니다.

{
  "script": {
    "inline": "doc['system.filesystem.free'].value / 1024 / 1024 /1024",
    "lang": "painless"
  }
}

좀 더 자세한 내용이 궁금 하신 분들은 아래 문서 한번 보시면 좋습니다.

 

공식문서)

https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html

 

 

:

[Kibana] Docker Compose 구성 하기

Elastic/Kibana 2020. 4. 2. 08:28

Kibana를 Docker로 구성 하기 위한 docker-compose.yml 내용을 살펴 봅니다.

 

참고문서)

 

https://www.elastic.co/guide/en/kibana/current/docker.html

 

docker-compose.yml)

version: '2.2'
services:
  ${KIBANA-SERVICE-NAME}:
    image: docker.elastic.co/kibana/kibana:7.6.2
    container_name: ${KIBANA-SERVICE-NAME}
#    depends_on:
#      - ${ES-SERVICE-NAME}
    ports:
      - 5601:5601
    expose:
      - 5601
    environment:
      ELASTICSEARCH_HOSTS: http://host.docker.internal:9200
    networks:
      - ${NETWORK-NAME}

networks:
  ${NETWORK-NAME}:
    driver: bridge

위 설정은 host 에서 Elasticsearch 를 Standalone 으로 띄우고 Kibana 를 컨테이너로 실행 시킨 후 연동 하도록 한 것입니다.

접속은 아래와 같이 하면 됩니다.

- http://localhost:5601

 

만약, docker-compose.yml 파일 내 Elasticsearch 와 Kibana 를 모두 구성해서 띄우실 때는 아래 설정을 맞춰 주면 됩니다.

- depends_on : 섹션에서 Elasticsearch 실행 후 Kibana 가 실행 되도록 구성

- network 구성은  같은 걸 사용하도록 하고 bridge 로 설정

- kibana 에서 elasticsearch 를 찾기 위해 host.docker.internal 이 아닌 ${ES-SERVICE-NAME} 으로 변경

 

또는 다른 인스턴스에 Elasticsearch 가 구성이 되어 있다면,

- environment: 섹션에서 host.docker.internal 이 아닌 Elasticsearch가 구성된 인스턴스의 DNS나 IP로 변경

하시면 됩니다.

 

 

 

 

:

[Kibana] Docker Compose 설정 시 environment 중

Elastic/Kibana 2020. 4. 1. 09:47

kibana docker compose 에서 아래 변수가 어떤 값을 참조 하는지 확인 하시기 바랍니다.

 

참고 문서)

https://www.elastic.co/guide/en/kibana/current/docker.html

environment:
    ELASTICSEARCH_URL: http://${SERVICE-NAME}:9200
    ELASTICSEARCH_HOSTS: http://${SERVICE-NAME}:9200
또는
    ELASTICSEARCH_URL: http://${CONTAINER-NAME}:9200
    ELASTICSEARCH_HOSTS: http://${CONTAINER-NAME}:9200

단일 노드에 Elasticsearch 와 Kibana 두 개의 컨테이너를 띄울 때 Kibana 에서는 Elasticsearch 의 Service 명 또는 Container 명으로 Elasticsearch 를 찾습니다.

:

[Elastic] 목차 입니다.

Elastic 2019. 11. 5. 14:52

Elastic Stack 의 Reference 목차 입니다.

이걸 왜 한 장으로 정리를 했냐면 목차만 잘 찾아 봐도 해결 방법이 어딨는지 어떤 기능을 제공 하고 있는지 쉽게 알수 있습니다.

(In my case!!)

 

그래서 혼자 보기 아까워서 그냥 올려봤습니다.

Elastic Stack References)

1. Elasticsearch

2. Logstash

3. Kibana

4. Beats Platform

5. Beats Developer Guide

6. Filebeat

더보기
  1. Elasticsearch

    1. Elasticsearch introduction

      1. Data in: documents and indices

      2. Information out: search and analyze

      3. Scalability and resilience

    2. Getting Started with Elasticsearch

      1. Get Elasticsearch up and running

      2. Index some documents

      3. Start searching

      4. Analyze results with aggregations

      5. Where to go from here

    3. Set up Elasticsearch

      1. Installing Elasticsearch

        1. Install Elasticsearch from archive on Linux or MacOS

        2. Install Elasticsearch with .zip on Windows

        3. Install Elasticsearch with Debian Package

        4. Install Elasticsearch with RPM

        5. Install Elasticsearch Windows MSI Installer

        6. Install Elasticsearch with Docker

        7. Install Elasticsearch on macOS with Homebrew

      2. Configuring Elasticsearch

        1. Setting JVM options

        2. Secure settings

        3. Logging configuration

        4. Auditing settings

        5. Cross-cluster replication settings

        6. Transforms settings

        7. Index lifecycle management settings

        8. License settings

        9. Machine learning settings

        10. Security settings

        11. SQL access settings

        12. Watcher settings

      3. Important Elasticsearch configuration

        1. path.data and path.logs

        2. cluster.name

        3. node.name

        4. network.host

        5. Discovery and cluster formation settings

        6. Setting the heap size

        7. JVM heap dump path

        8. GC logging

        9. Temp directory

        10. JVM fatal error logs

      4. Important System Configuration

        1. Configuring system settings

        2. Disable swapping

        3. File Descriptors

        4. Virtual memory

        5. Number of threads

        6. DNS cache settings

        7. JNA temporary directory not mounted with noexec

      5. Bootstrap Checks

        1. Heap size check

        2. File descriptor check

        3. Memory lock check

        4. Maximum number of threads check

        5. Max file size check

        6. Maximum size virtual memory check

        7. Maximum map count check

        8. Client JVM check

        9. Use serial collector check

        10. System call filter check

        11. OnError and OnOutOfMemoryError checks

        12. Early-access check

        13. G1GC check

        14. All permission check

        15. Discovery configuration check

      6. Starting Elasticsearch

      7. Stopping Elasticsearch

      8. Adding nodes to your cluster

      9. Set up X-Pack

      10. Configuring X-Pack Java Clients

      11. Bootstrap Checks for X-Pack

    4. Upgrade Elasticsearch

      1. Rolling upgrades

      2. Full cluster restart upgrade

      3. Reindex before upgrading

        1. Reindex in place

        2. Reindex from a remote cluster

    5. Aggregations

      1. Metrics Aggregations

        1. Avg Aggregation

        2. Weighted Avg Aggregation

        3. Cardinality Aggregation

        4. Extended Stats Aggregation

        5. Geo Bounds Aggregation

        6. Geo Centroid Aggregation

        7. Max Aggregation

        8. Min Aggregation

        9. Percentiles Aggregation

        10. Percentile Ranks Aggregation

        11. Scripted Metric Aggregation

        12. Stats Aggregation

        13. Sum Aggregation

        14. Top Hits Aggregation

        15. Value Count Aggregation

        16. Median Absolute Deviation Aggregation

      2. Bucket Aggregations

        1. Adjacency Matrix Aggregation

        2. Auto-interval Date Histogram Aggregation

        3. Children Aggregation

        4. Composite Aggregation

        5. Date Histogram Aggregation

        6. Date Range Aggregation

        7. Diversified Sampler Aggregation

        8. Filter Aggregation

        9. Filters Aggregation

        10. Geo Distance Aggregation

        11. GeoHash grid Aggregation

        12. GeoTile Grid Aggregation

        13. Global Aggregation

        14. Histogram Aggregation

        15. IP Range Aggregation

        16. Missing Aggregation

        17. Parent Aggregation

        18. Range Aggregation

        19. Rare Terms Aggregation

        20. Reverse nested Aggregation

        21. Sampler Aggregation

        22. Significant Terms Aggregation

        23. Significant Text Aggregation

        24. Terms Aggregation

        25. Subtleties of bucketing range fields

      3. Pipeline Aggregations

        1. Avg Bucket Aggregation

        2. Derivative Aggregation

        3. Max Bucket Aggregation

        4. Min Bucket Aggregation

        5. Sum Bucket Aggregation

        6. Stats Bucket Aggregation

        7. Extended Stats Bucket Aggregation

        8. Percentiles Bucket Aggregation

        9. Moving Average Aggregation

        10. Moving Function Aggregation

        11. Cumulative Sum Aggregation

        12. Cumulative Cardinality Aggregation

        13. Bucket Script Aggregation

        14. Bucket Selector Aggregation

        15. Bucket Sort Aggregation

        16. Serial Differencing Aggregation

      4. Matrix Aggregations

        1. Matrix Stats

      5. Caching heavy aggregations

      6. Returning only aggregations

      7. Aggregation Metadata

      8. Returning the type of the aggregation

    6. Query DSL

      1. Query and filter context

      2. Compound queries

        1. Boolean

        2. Boosting

        3. Constant score

        4. Disjunction score

        5. Function score

      3. Full text queries

        1. Intervals

        2. Match

        3. Match boolean prefix

        4. Match phrase

        5. Match phrase prefix

        6. Multi-match

        7. Common Terms Query

        8. Query String

        9. Simple query string

      4. Geo queries

        1. Geo-bounding box

        2. Geo-distance

        3. Geo-polygon

        4. Geo-shape

      5. Shape queries

        1. Shape

      6. Joining queries

        1. Nested

        2. Has child

        3. Has parent

        4. Parent ID

      7. Match all

      8. Span queries

        1. Span containing

        2. Span field masking

        3. Span first

        4. Span multi-term

        5. Span near

        6. Span not

        7. Span or

        8. Span term

        9. Span within

      9. Specialized queries

        1. Distance feature

        2. More like this

        3. Percolate

        4. Rank feature

        5. Script

        6. Script score

        7. Wrapper

        8. Pinned Query

      10. Term-level queries

        1. Exists

        2. Fuzzy

        3. IDs

        4. Prefix

        5. Range

        6. Regexp

        7. Term

        8. Terms

        9. Terms set

        10. Type Query

        11. Wildcard

      11. minimum_should_match parameter

      12. rewrite parameter

      13. Regular expression syntax

    7. Search across clusters

    8. Scripting

      1. How to use scripts

      2. Accessing document fields and special variables

      3. Scripting and security

      4. Painless scripting language

      5. Lucene expressions language

      6. Advanced scripts using script engines

    9. Mapping

      1. Removal of mapping types

      2. Field datatypes

        1. Alias

        2. Arrays

        3. Binary

        4. Boolean

        5. Date

        6. Date nanoseconds

        7. Dense vector

        8. Flattened

        9. Geo-point

        10. Geo-shape

        11. IP

        12. Join

        13. Keyword

        14. Nested

        15. Numeric

        16. Object

        17. Percolator

        18. Range

        19. Rank feature

        20. Rank features

        21. Search-as-you-type

        22. Sparse vector

        23. Text

        24. Token count

        25. Shape

      3. Meta-Fields

        1. _field_names field

        2. _ignored field

        3. _id field

        4. _index field

        5. _meta field

        6. _routing field

        7. _source field

        8. _type field

      4. Mapping parameters

        1. analyzer

        2. normalizer

        3. boost

        4. coerce

        5. copy_to

        6. doc_values

        7. dynamic

        8. enabled

        9. eager_global_ordinals

        10. fielddata

        11. format

        12. ignore_above

        13. ignore_malformed

        14. index

        15. index_options

        16. index_phrases

        17. index_prefixes

        18. fields

        19. norms

        20. null_value

        21. position_increment_gap

        22. properties

        23. search_analyzer

        24. similarity

        25. store

        26. term_vector

      5. Dynamic Mapping

        1. Dynamic field mapping

        2. Dynamic templates

    10. Analysis

      1. Anatomy of an analyzer

      2. Testing analyzers

      3. Analyzers

        1. Configuring built-in analyzers

        2. Fingerprint Analyzer

        3. Keyword Analyzer

        4. Language Analyzers

        5. Pattern Analyzer

        6. Simple Analyzer

        7. Standard Analyzer

        8. Stop Analyzer

        9. Whitespace Analyzer

        10. Custom Analyzer

      4. Normalizers

      5. Tokenizers

        1. Char Group Tokenizer

        2. Classic Tokenizer

        3. Edge NGram Tokenizer

        4. Keyword Tokenizer

        5. Letter Tokenizer

        6. Lowercase Tokenizer

        7. NGram Tokenizer

        8. Path Hierachy Tokenizer

        9. Pattern Tokenizer

        10. Simple Pattern Tokenizer

        11. Simple Pattern Split Tokenizer

        12. Standard Tokenizer

        13. Thai Tokenizer

        14. UAX URL Email Tokenizer

        15. Whitespace Tokenizer

      6. Token Filters

        1. Apostrophe

        2. ASCII Folding Token Filter

        3. CJK bigram

        4. CJK width

        5. Classic Token Filter

        6. Common Grams Token Filter

        7. Compound Word Token Filters

        8. Conditional Token Filter

        9. Decimal Digit Token Filter

        10. Delimited Payload Token Filter

        11. Edge NGram Token Filter

        12. Elision Token Filter

        13. Fingerprint Token Filter

        14. Flatten Graph Token Filter

        15. Hunspell Token Filter

        16. Keep Types Token Filter

        17. Keep Words Token Filter

        18. Keyword Request Token Filter

        19. KStem Token Filter

        20. Length Token Filter

        21. Limit Token Count Token Filter

        22. Lowercase Token Filter

        23. MinHash Token Filter

        24. Multiplexer Token Filter

        25. NGram Token Filter

        26. Normalization Token Filter

        27. Pattern Capture Token Filter

        28. Pattern Replace Token Filter

        29. Phonetic Token Filter

        30. Porter Stem Token Filter

        31. Predicate Token Filter Script

        32. Remove Duplicates Token Filter

        33. Reverse Token Filter

        34. Shingle Token Filter

        35. Snowball Token Filter

        36. Stemmer Token Filter

        37. Stemmer Override Token Filter

        38. Stop Token Filter

        39. Synonym Token Filter

        40. Synonym Graph Token Filter

        41. Trim Token Filter

        42. Truncate Token Filter

        43. Unique Token Filter

        44. Uppercase Token Filter

        45. Word Delimiter Token Filter

        46. Word Delimiter Graph Token Filter

      7. Character Filters

        1. HTML Strip Char Filter

        2. Mapping Char Filter

        3. Pattern Replace Char Filter

    11. Modules

      1. Discovery and cluster formation

        1. Discovery

        2. Quorum-based decision making

        3. Voting configurations

        4. Bootstrapping a cluster

        5. Adding and removing nodes

        6. Publishing the cluster state

        7. Cluster fault detection

        8. Discovery and cluster formation settings

      2. Shard allocation and cluster-level routing

        1. Cluster level shard allocation

        2. Disk-based shard allocation

        3. Shard allocation awareness

        4. Cluster-level shard allocation filtering

        5. Miscellaneous cluster settings

      3. Local Gateway

        1. Dangling indices

      4. HTTP

      5. Indices

        1. Circuit Breaker

        2. Fielddata

        3. Node Query Cache

        4. Indexing Buffer

        5. Shard request cache

        6. Index recovery

        7. Search Settings

      6. Network Settings

      7. Node

      8. Plugins

      9. Snapshot And Restore

      10. Thread Pool

      11. Transport

      12. Remote clusters

    12. Index modules

      1. Analysis

      2. Index Shard Allocation

        1. Index-level shard allocation filtering

        2. Delaying allocation when a node leaves

        3. Index recovery prioritization

        4. Total shards per node

      3. Mapper

      4. Merge

      5. Similarity module

      6. Slow Log

      7. Store

        1. Preloading data into the file system cache

      8. Translog

      9. History retention

      10. Index Sorting

        1. Use index sorting to speed up conjunctions

    13. Ingest node

      1. Pipeline Definition

      2. Accessing Data in Pipelines

      3. Conditional Execution in Pipelines

        1. Handling Nested Fields in Conditionals

        2. Complex Conditionals

        3. Conditionals with the Pipeline Processor

        4. Conditionals with the Regular Expressions

      4. Handling Failures in Pipelines

      5. Processors

        1. Append Processor

        2. Bytes Processor

        3. Circle Processor

        4. Convert Processor

        5. Date Processor

        6. Date Index Name Processor

        7. Dissect Processor

        8. Dot Expander Processor

        9. Drop Processor

        10. Fail Processor

        11. Foreach Processor

        12. GeoIP Processor

        13. Grok Processor

        14. Gsub Processor

        15. HTML Strip Processor

        16. Join Processor

        17. JSON Processor

        18. KV Processor

        19. Lowercase Processor

        20. Pipeline Processor

        21. Remove Processor

        22. Rename Processor

        23. Script Processor

        24. Set Processor

        25. Set Security User Processor

        26. Split Processor

        27. Sort Processor

        28. Trim Processor

        29. Uppercase Processor

        30. URL Decode Processor

        31. User Agent processor

    14. Managing the index lifecycle

      1. Getting started with index lifecycle management

      2. Policy phases and actions

        1. Timing

        2. Phase Execution

        3. Actions

        4. Full Policy

      3. Set up index lifecycle management policy

        1. Applying a policy to an index template

        2. Apply a policy to a create index request

      4. Using policies to manage index rollover

        1. Skipping Rollover

      5. Update policy

        1. Updates to policies not managing indices

        2. Updates to executing policies

        3. Switching policies for an index

      6. Index lifecycle error handling

      7. Restoring snapshots of managed indices

      8. Start and stop index lifecycle management

      9. Using ILM with existing indices

        1. Managing existing periodic indices with ILM

        2. Reindexing via ILM

      10. Getting started with snapshot lifecycle management

    15. SQL access

      1. Overview

      2. Getting Started with SQL

      3. Conventions and Terminology

        1. Mapping concepts across SQL and Elasticsearch

      4. Security

      5. SQL REST API

        1. Overview

        2. Response Data Formats

        3. Paginating through a large response

        4. Filtering using Elasticsearch query DSL

        5. Columnar results

        6. Supported REST parameters

      6. SQL Translate API

      7. SQL CLI

      8. SQL JDBC

        1. API usage

      9. SQL ODBC

        1. Driver installation

        2. Configuration

      10. SQL Client Applications

        1. DBeaver

        2. DbVisualizer

        3. Microsoft Excel

        4. Microsoft Power BI Desktop

        5. Microsoft PowerShell

        6. MicroStrategy Desktop

        7. Qlik Sense Desktop

        8. SQuirreL SQL

        9. SQL Workbench/J

        10. Tableau Desktop

      11. SQL Language

        1. Lexical Structure

        2. SQL Commands

        3. DESCRIBE TABLE

        4. SELECT

        5. SHOW COLUMNS

        6. SHOW FUNCTIONS

        7. SHOW TABLES

        8. Data Types

        9. Index patterns

        10. Frozen Indices

      12. Functions and Operators

        1. Comparison Operators

        2. Logical Operators

        3. Math Operators

        4. Cast Operators

        5. LIKE and RLIKE Operators

        6. Aggregate Functions

        7. Grouping Functions

        8. Date/Time and Interval Functions and Operators

        9. Full-Text Search Functions

        10. Mathematical Functions

        11. String Functions

        12. Type Conversion Functions

        13. Geo Functions

        14. Conditional Functions And Expressions

        15. System Functions

      13. Reserved keywords

      14. SQL Limitations

    16. Monitor a cluster

      1. Overview

      2. How it works

      3. Monitoring in a production environment

      4. Collecting monitoring data

        1. Pausing data collection

      5. Collecting monitoring data with Metricbeat

      6. Collecting log data with Filebeat

      7. Configuring indices for monitoring

      8. Collectors

      9. Exporters

        1. Local exporters

        2. HTTP exporters

      10. Troubleshooting

    17. Frozen indices

      1. Best practices

      2. Searching a frozen index

      3. Monitoring frozen indices

    18. Roll up or transform your data

      1. Rolling up historical data

        1. Overview

        2. API quick reference

        3. Getting started

        4. Understanding groups

        5. Rollup aggregation limitations

        6. Rollup search limitations

      2. Transforming data

        1. Overview

        2. When to use transforms

        3. How checkpoints work

        4. API quick reference

        5. Tutorial: Transforming the eCommerce sample data

        6. Examples

        7. Troubleshooting

        8. Limitations

    19. Set up a cluster for high availability

      1. Back up a cluster

        1. Back up the data

        2. Back up the cluster configuration

        3. Back up the security configuration

        4. Restore the security configuration

        5. Restore the data

      2. Cross-cluster replication

        1. Overview

        2. Requirements for leader indices

        3. Automatically following indices

        4. Getting started with cross-cluster replication

        5. Remote recovery

        6. Upgrading clusters

    20. Secure a cluster

      1. Overview

      2. Configuring security

        1. Encrypting communications in Elasticsearch

        2. Encrypting communications in an Elasticsearch Docker Container

        3. Enabling cipher suites for stronger encryption

        4. Separating node-to-node and client traffic

        5. Configuring an Active Directory realm

        6. Configuring a file realm

        7. Configuring an LDAP realm

        8. Configuring a native realm

        9. Configuring a PKI realm

        10. Configuring a SAML realm

        11. Configuring a Kerberos realm

        12. Security files

        13. FIPS 140-2

      3. How security works

      4. User authentication

        1. Built-in users

        2. Internal users

        3. Realms

        4. Realm chains

        5. Active Directory user authentication

        6. File-based user authentication

        7. LDAP user authentication

        8. Native user authentication

        9. PKI user authentication

        10. SAML authentication

        11. Kerberos authentication

        12. Integrating with other authentication systems

        13. Enabling anonymous access

        14. Controlling the user cache

      5. Configuring SAML single-sign-on on the Elastic Stack

        1. The identity provider

        2. Configure Elasticsearch for SAML authentication

        3. Generating SP metadata

        4. Configuring role mappings

        5. User metadata

        6. Configuring Kibana

        7. Troubleshooting SAML Realm Configuration

      6. Configuring single sign-on to the Elastic Stack using OpenID Connect

        1. The OpenID Connect Provider

        2. Configure Elasticsearch for OpenID Connect authentication

        3. Configuring role mappings

        4. User metadata

        5. Configuring Kibana

        6. OpenID Connect without Kibana

      7. User authorization

        1. Built-in roles

        2. Defining roles

        3. Security privileges

        4. Document level security

        5. Field level security

        6. Granting privileges for indices and aliases

        7. Mapping users and groups to roles

        8. Setting up field and document level security

        9. Submitting requests on behalf of other users

        10. Configuring authorization delegation

        11. Customizing roles and authorization

      8. Auditing security events

        1. Audit event types

        2. Logfile audit output

        3. Auditing search queries

      9. Encrypting communications

        1. Setting up TLS on a cluster

      10. Restricting connections with IP filtering

      11. Cross cluster search, clients, and integrations

        1. Cross cluster search and security

        2. Java Client and security

        3. HTTP/REST clients and security

        4. ES-Hadoop and Security

        5. Beats and Security

        6. Monitoring and security

      12. Tutorial: Getting started with security

        1. Enable Elasticsearch security features

        2. Create passwords for built-in users

        3. Add the built-in user to Kibana

        4. Configure authentication

        5. Create users

        6. Assign roles

        7. Add user information in Logstash

        8. View system metrics in Kibana

      13. Tutorial: Encrypting communications

        1. Generate certificates

        2. Encrypt internode communications

        3. Add nodes to your cluster

      14. Troubleshooting

        1. Some settings are not returned via the nodes settings API

        2. Authorization exceptions

        3. Users command fails due to extra arguments

        4. Users are frequently locked out of Active Directory

        5. Certificate verification fails for curl on Mac

        6. SSLHandshakeException causes connections to fail

        7. Common SSL/TLS exceptions

        8. Common Kerberos exceptions

        9. Common SAML issues

        10. Internal Server Error in Kibana

        11. Setup-passwords command fails due to connection failure

        12. Failures due to relocation of the configuration files

      15. Limitations

    21. Alerting on cluster and index events

      1. Getting started with Watcher

      2. How Watcher works

      3. Encrypting sensitive data in Watcher

      4. Inputs

        1. Simple input

        2. Search input

        3. HTTP input

        4. Chain input

      5. Triggers

        1. Schedule trigger

      6. Conditions

        1. Always condition

        2. Never condition

        3. Compare condition

        4. Array compare condition

        5. Script condition

      7. Actions

        1. Running an action for each element in an array

        2. Adding conditions to actions

        3. Email action

        4. Webhook action

        5. Index action

        6. Logging Action

        7. Slack Action

        8. PagerDuty action

        9. Jira action

      8. Transforms

        1. Search transform

        2. Script transform

        3. Chain transform

      9. Java API

      10. Managing watches

      11. Example watches

        1. Watching the status of an Elasticsearch cluster

        2. Watching event data

      12. Troubleshooting

      13. Limitations

    22. Command line tools

      1. elasticsearch-certgen

      2. elasticsearch-certutil

      3. elasticsearch-croneval

      4. elasticsearch-migrate

      5. elasticsearch-node

      6. elasticsearch-saml-metadata

      7. elasticsearch-setup-passwords

      8. elasticsearch-shard

      9. elasticsearch-syskeygen

      10. elasticsearch-users

    23. How To

      1. General recommendations

      2. Recipes

        1. Mixing exact search with stemming

        2. Getting consistent scoring

        3. Incorporating static relevance signals into the score

      3. Tune for indexing speed

      4. Tune for search speed

        1. Tune your queries with the Profile API

        2. Faster phrase queries with index_phrases

        3. Faster prefix queries with index_prefixes

      5. Tune for disk usage

    24. Testing

      1. Java Testing Framework

        1. Why randomized testing?

        2. Using the Elasticsearch test classes

        3. Unit tests

        4. Integration tests

        5. Randomized testing

        6. Assertions

    25. Glossary of terms

    26. REST APIs

      1. API conventions

        1. Multiple Indices

        2. Date math support in index names

        3. Common options

        4. URL-based access control

      2. cat APIs

        1. cat aliases

        2. cat allocation

        3. cat count

        4. cat fielddata

        5. cat health

        6. cat indices

        7. cat master

        8. cat nodeattrs

        9. cat nodes

        10. cat pending tasks

        11. cat plugins

        12. cat recovery

        13. cat repositories

        14. cat task management

        15. cat thread pool

        16. cat shards

        17. cat segments

        18. cat snapshots

        19. cat templates

      3. Cluster APIs

        1. Cluster Health

        2. Cluster State

        3. Cluster Stats

        4. Pending cluster tasks

        5. Cluster Reroute

        6. Cluster Update Settings

        7. Cluster Get Settings

        8. Nodes Stats

        9. Nodes Info

        10. Nodes Feature Usage

        11. Remote Cluster Info

        12. Task management

        13. Nodes hot_threads

        14. Cluster Allocation Explain API

        15. Voting Configuration Exclusions

      4. Cross-cluster replication APIs

        1. Get CCR stats

        2. Create follower

        3. Pause follower

        4. Resume follower

        5. Unfollow

        6. Forget follower

        7. Get follower stats

        8. Get follower info

        9. Create auto-follow pattern

        10. Delete auto-follow pattern

        11. Get auto-follow pattern

      5. Document APIs

        1. Reading and Writing documents

        2. Index

        3. Get

        4. Delete

        5. Delete by query

        6. Update

        7. Update By Query API

        8. Multi get

        9. Bulk

        10. Reindex

        11. Term vectors

        12. Multi term vectors

        13. ?refresh

        14. Optimistic concurrency control

      6. Explore API

      7. Index APIs

        1. Add index alias

        2. Analyze

        3. Clear cache

        4. Clone index

        5. Close index

        6. Create index

        7. Delete index

        8. Delete index alias

        9. Delete index template

        10. Flush

        11. Force merge

        12. Freeze index

        13. Get field mapping

        14. Get index

        15. Get index alias

        16. Get index settings

        17. Get index template

        18. Get mapping

        19. Index alias exists

        20. Index exists

        21. Index recovery

        22. Index segments

        23. Index shard stores

        24. Index stats

        25. Index template exists

        26. Open index

        27. Put index template

        28. Put mapping

        29. Refresh

        30. Rollover index

        31. Shrink index

        32. Split index

        33. Synced flush

        34. Type exists

        35. Unfreeze index

        36. Update index alias

        37. Update index settings

      8. Index lifecycle management API

        1. Create policy

        2. Get policy

        3. Delete policy

        4. Move to step

        5. Remove policy

        6. Retry policy

        7. Get index lifecycle management status

        8. Explain lifecycle

        9. Start index lifecycle management

        10. Stop index lifecycle management

      9. Ingest APIs

        1. Put pipeline

        2. Get pipeline

        3. Delete pipeline

        4. Simulate pipeline

      10. Info API

      11. Licensing APIs

        1. Delete license

        2. Get license

        3. Get trial status

        4. Start trial

        5. Get basic status

        6. Start basic

        7. Update license

      12. Machine learning anomaly detection APIs

        1. Add events to calendar

        2. Add jobs to calendar

        3. Close jobs

        4. Create jobs

        5. Create calendar

        6. Create datafeeds

        7. Create filter

        8. Delete calendar

        9. Delete datafeeds

        10. Delete events from calendar

        11. Delete filter

        12. Delete forecast

        13. Delete jobs

        14. Delete jobs from calendar

        15. Delete model snapshots

        16. Delete expired data

        17. Find file structure

        18. Flush jobs

        19. Forecast jobs

        20. Get buckets

        21. Get calendars

        22. Get categories

        23. Get datafeeds

        24. Get datafeed statistics

        25. Get influencers

        26. Get jobs

        27. Get job statistics

        28. Get machine learning info

        29. Get model snapshots

        30. Get overall buckets

        31. Get scheduled events

        32. Get filters

        33. Get records

        34. Open jobs

        35. Post data to jobs

        36. Preview datafeeds

        37. Revert model snapshots

        38. Set upgrade mode

        39. Start datafeeds

        40. Stop datafeeds

        41. Update datafeeds

        42. Update filter

        43. Update jobs

        44. Update model snapshots

      13. Machine learning data frame analytics APIs

        1. Create data frame analytics jobs

        2. Delete data frame analytics jobs

        3. Evaluate data frame analytics

        4. Estimate memory usage for data frame analytics jobs

        5. Get data frame analytics jobs

        6. Get data frame analytics jobs stats

        7. Start data frame analytics jobs

        8. Stop data frame analytics jobs

      14. Migration APIs

        1. Deprecation info

      15. Reload search analyzers

      16. Rollup APIs

        1. Create rollup jobs

        2. Delete rollup jobs

        3. Get job

        4. Get rollup caps

        5. Get rollup index caps

        6. Rollup search

        7. Rollup job configuration

        8. Start rollup jobs

        9. Stop rollup jobs

      17. Search APIs

        1. Search

        2. URI Search

        3. Request Body Search

        4. Search Template

        5. Multi Search Template

        6. Search Shards API

        7. Suggesters

        8. Multi Search API

        9. Count API

        10. Validate API

        11. Explain API

        12. Profile API

        13. Field Capabilities API

        14. Ranking Evaluation API

      18. Security APIs

        1. Authenticate

        2. Change passwords

        3. Clear cache

        4. Clear roles cache

        5. Create API keys

        6. Create or update application privileges

        7. Create or update role mappings

        8. Create or update roles

        9. Create or update users

        10. Delegate PKI authentication

        11. Delete application privileges

        12. Delete role mappings

        13. Delete roles

        14. Delete users

        15. Disable users

        16. Enable users

        17. Get API key information

        18. Get application privileges

        19. Get builtin privileges

        20. Get role mappings

        21. Get roles

        22. Get token

        23. Get users

        24. Has privileges

        25. Invalidate API key

        26. Invalidate token

        27. OpenID Connect Prepare Authentication API

        28. OpenID Connect authenticate API

        29. OpenID Connect logout API

        30. SSL certificate

      19. Snapshot lifecycle management API

        1. Put snapshot lifecycle policy

        2. Get snapshot lifecycle policy

        3. Execute snapshot lifecycle policy

        4. Delete snapshot lifecycle policy

      20. Transform APIs

        1. Create transforms

        2. Update transforms

        3. Delete transforms

        4. Get transforms

        5. Get transform statistics

        6. Preview transforms

        7. Start transforms

        8. Stop transforms

      21. Watcher APIs

        1. Ack watch

        2. Activate watch

        3. Deactivate watch

        4. Delete watch

        5. Execute watch

        6. Get watch

        7. Get Watcher stats

        8. Put watch

        9. Start watch service

        10. Stop watch service

      22. Definitions

        1. Datafeed resources

        2. Data frame analytics job resources

        3. Data frame analytics evaluation resources

        4. Job resources

        5. Job statistics

        6. Model snapshot resources

        7. Role mapping resources

        8. Results resources

        9. Transform resources

  2. Logstash

    1. Logstash Introduction

    2. Getting Started with Logstash

      1. Installing Logstash

      2. Stashing Your First Event

      3. Parsing Logs with Logstash

      4. Stitching Together Multiple Input and Output Plugins

    3. How Logstash Works

      1. Execution Model

    4. Setting Up and Running Logstash

      1. Logstash Directory Layout

      2. Logstash Configuration Files

      3. logstash.yml

      4. Secrets keystore for secure settings

      5. Running Logstash from the Command Line

      6. Running Logstash as a Service on Debian or RPM

      7. Running Logstash on Docker

      8. Configuring Logstash for Docker

      9. Running Logstash on Windows

      10. Logging

      11. Shutting Down Logstash

      12. Setting Up X-Pack

    5. Upgrading Logstash

      1. Upgrading Using Package Managers

      2. Upgrading Using a Direct Download

      3. Upgrading between minor versions

      4. Upgrading Logstash to 7.0

      5. Upgrading with the Persistent Queue Enabled

    6. Configuring Logstash

      1. Structure of a Config File

      2. Accessing Event Data and Fields in the Configuration

      3. Using Environment Variables in the Configuration

      4. Logstash Configuration Examples

      5. Multiple Pipelines

      6. Pipeline-to-Pipeline Communication

      7. Reloading the Config File

      8. Managing Multiline Events

      9. Glob Pattern Support

      10. Converting Ingest Node Pipelines

      11. Logstash-to-Logstash Communication

      12. Centralized Pipeline Management

      13. X-Pack security

      14. X-Pack Settings

    7. Managing Logstash

      1. Centralized Pipeline Management

    8. Working with Logstash Modules

      1. Using Elastic Cloud

      2. ArcSight Module

      3. Netflow Module (deprecated)

      4. Azure Module

    9. Working with Filebeat Modules

      1. Use ingest pipelines for parsing

      2. Use Logstash pipelines for parsing

      3. Example: Set up Filebeat modules to work with Kafka and Logstash

    10. Data Resiliency

      1. Persistent Queues

      2. Dead Letter Queues

    11. Transforming Data

      1. Performing Core Operations

      2. Deserializing Data

      3. Extracting Fields and Wrangling Data

      4. Enriching Data with Lookups

    12. Deploying and Scaling Logstash

    13. Performance Tuning

      1. Performance Troubleshooting Guide

      2. Tuning and Profiling Logstash Performance

    14. Monitoring Logstash with APIs

      1. Node Info API

      2. Plugins Info API

      3. Node Stats API

      4. Hot Threads API

    15. Monitoring Logstash with X-Pack

      1. Metricbeat collection

      2. Internal collection

      3. Monitoring UI

      4. Pipeline Viewer UI

      5. Troubleshooting

    16. Working with plugins

      1. Generating Plugins

      2. Offline Plugin Management

      3. Private Gem Repositories

      4. Event API

    17. Input plugins

      1. azure_event_hubs

      2. beats

      3. cloudwatch

      4. couchdb_changes

      5. dead_letter_queue

      6. elasticsearch

      7. exec

      8. file

      9. ganglia

      10. gelf

      11. generator

      12. github

      13. google_cloud_storage

      14. google_pubsub

      15. graphite

      16. heartbeat

      17. http

      18. http_poller

      19. imap

      20. irc

      21. java_generator

      22. java_stdin

      23. jdbc

      24. jms

      25. jmx

      26. kafka

      27. kinesis

      28. log4j

      29. lumberjack

      30. meetup

      31. pipe

      32. puppet_facter

      33. rabbitmq

      34. redis

      35. relp

      36. rss

      37. s3

      38. salesforce

      39. snmp

      40. snmptrap

      41. sqlite

      42. sqs

      43. stdin

      44. stomp

      45. syslog

      46. tcp

      47. twitter

      48. udp

      49. unix

      50. varnishlog

      51. websocket

      52. wmi

      53. xmpp

    18. Output plugins

      1. boundary

      2. circonus

      3. cloudwatch

      4. csv

      5. datadog

      6. datadog_metrics

      7. elastic_app_search

      8. elasticsearch

      9. email

      10. exec

      11. file

      12. ganglia

      13. gelf

      14. google_bigquery

      15. google_cloud_storage

      16. google_pubsub

      17. graphite

      18. graphtastic

      19. http

      20. influxdb

      21. irc

      22. java_sink

      23. java_stdout

      24. juggernaut

      25. kafka

      26. librato

      27. loggly

      28. lumberjack

      29. metriccatcher

      30. mongodb

      31. nagios

      32. nagios_nsca

      33. opentsdb

      34. pagerduty

      35. pipe

      36. rabbitmq

      37. redis

      38. redmine

      39. riak

      40. riemann

      41. s3

      42. sns

      43. solr_http

      44. sqs

      45. statsd

      46. stdout

      47. stomp

      48. syslog

      49. tcp

      50. timber

      51. udp

      52. webhdfs

      53. websocket

      54. xmpp

      55. zabbix

    19. Filter plugins

      1. aggregate

      2. alter

      3. bytes

      4. cidr

      5. cipher

      6. clone

      7. csv

      8. date

      9. de_dot

      10. dissect

      11. dns

      12. drop

      13. elapsed

      14. elasticsearch

      15. environment

      16. extractnumbers

      17. fingerprint

      18. geoip

      19. grok

      20. http

      21. i18n

      22. java_uuid

      23. jdbc_static

      24. jdbc_streaming

      25. json

      26. json_encode

      27. kv

      28. memcached

      29. metricize

      30. metrics

      31. mutate

      32. prune

      33. range

      34. ruby

      35. sleep

      36. split

      37. syslog_pri

      38. threats_classifier

      39. throttle

      40. tld

      41. translate

      42. truncate

      43. urldecode

      44. useragent

      45. uuid

      46. xml

    20. Codec plugins

      1. avro

      2. cef

      3. cloudfront

      4. cloudtrail

      5. collectd

      6. dots

      7. edn

      8. edn_lines

      9. es_bulk

      10. fluent

      11. graphite

      12. gzip_lines

      13. jdots

      14. java_line

      15. java_plain

      16. json

      17. json_lines

      18. line

      19. msgpack

      20. multiline

      21. netflow

      22. nmap

      23. plain

      24. protobuf

      25. rubydebug

    21. Tips and Best Practices

    22. Troubleshooting Common Problems

    23. Contributing to Logstash

      1. How to write a Logstash input plugin

      2. How to write a Logstash codec plugin

      3. How to write a Logstash filter plugin

      4. How to write a Logstash output plugin

      5. Documenting your plugin

      6. Contributing a Patch to a Logstash Plugin

      7. Logstash Plugins Community Maintainer Guide

      8. Submitting your plugin to RubyGems.org and the logstash-plugins repository

    24. Contributing a Java Plugin

      1. How to write a Java input plugin

      2. How to write a Java codec plugin

      3. How to write a Java filter plugin

      4. How to write a Java output plugin

    25. Glossary of Terms

  3. Kibana

    1. Introduction

    2. Set Up Kibana

      1. Installing Kibana

      2. Install Kibana with .tar.gz

      3. Install Kibana with Debian Package

      4. Install Kibana with RPM

      5. Install Kibana on Windows

      6. Install Kibana on macOS with Homebrew

    3. Starting and stopping Kibana

    4. Configuring Kibana

      1. APM settings

      2. Code settings

      3. Development tools settings

      4. Graph settings

      5. Infrastructure UI settings

      6. i18n settings in Kibana

      7. Logs UI settings

      8. Machine learning settings

      9. Monitoring settings

      10. Reporting settings

      11. Secure settings

      12. Security settings

      13. Spaces settings

    5. Running Kibana on Docker

    6. Accessing Kibana

    7. Connect Kibana with Elasticsearch

    8. Using Kibana in a production environment

    9. Upgrading Kibana

      1. Standard upgrade

      2. Troubleshooting saved object migrations

    10. Configuring monitoring

      1. Collecting monitoring data

      2. Collecting monitoring data with Metricbeat

      3. Viewing monitoring data

    11. Configuring security

      1. Authentication

      2. Encrypting communications

      3. Audit Logging

    12. Getting Started

      1. Add sample data

      2. Explore Kibana using sample data

      3. Build your own dashboard

        1. Define your index patterns

        2. Discover your data

        3. Visualize your data

        4. Add visualizations to a dashboard

    13. Discover

      1. Setting the time filter

      2. Searching your data

        1. Kibana Query Language

        2. Lucene query syntax

        3. Saving searches

        4. Saving queries

        5. Change the indices you’re searching

        6. Refresh the search results

      3. Filtering by Field

      4. Viewing Document Data

      5. Viewing Document Context

      6. Viewing Field Data Statistics

    14. Visualize

      1. Creating a Visualization

      2. Saving Visualizations

      3. Using rolled up data in a visualization

      4. Line, Area, and Bar charts

      5. Controls Visualization

        1. Adding Input Controls

        2. Global Options

      6. Data Table

      7. Markdown Widget

      8. Metric

      9. Goal and Gauge

      10. Pie Charts

      11. Coordinate Maps

      12. Region Maps

      13. Timelion

      14. TSVB

      15. Tag Clouds

      16. Heatmap Chart

      17. Vega Graphs

        1. Getting Started with Vega

        2. Vega vs Vega-Lite

        3. Querying Elasticsearch

        4. Elastic Map Files

        5. Vega with a Map

        6. Debugging

        7. Useful Links

      18. Inspecting Visualizations

    15. Dashboard

      1. Create a dashboard

      2. Dashboard-only mode

    16. Canvas

      1. Canvas tutorial

      2. Create a workpad

      3. Showcase your data with elements

      4. Present your workpad

      5. Share your workpad

      6. Canvas function reference

        1. TinyMath functions

    17. Extend your use case

      1. Graph data connections

        1. Using Graph

        2. Configuring Graph

        3. Troubleshooting

        4. Limitations

      2. Machine learning

    18. Elastic Maps

      1. Getting started with Elastic Maps

        1. Creating a new map

        2. Adding a choropleth layer

        3. Adding layers for Elasticsearch data

        4. Saving the map

        5. Adding the map to a dashboard

      2. Heat map layer

      3. Tile layer

      4. Vector layer

        1. Vector styling

        2. Vector style properties

        3. Vector tooltips

      5. Plot big data without plotting too much data

        1. Grid aggregation

        2. Most recent entities

        3. Point to point

        4. Term join

      6. Searching your data

        1. Creating filters from your map

        2. Filtering a single layer

        3. Searching across multiple indices

      7. Connecting to Elastic Maps Service

      8. Upload GeoJSON data

      9. Indexing GeoJSON data tutorial

      10. Elastic Maps troubleshooting

    19. Code

      1. Import your first repo

      2. Repo management

      3. Install language server

      4. Basic navigation

      5. Semantic code navigation

      6. Search

      7. Config for multiple Kibana instances

    20. Infrastructure

      1. Getting started with infrastructure monitoring

      2. Using the Infrastructure app

      3. Viewing infrastructure metrics

      4. Metrics Explorer

    21. Logs

      1. Getting started with logs monitoring

      2. Using the Logs app

      3. Configuring the Logs data

    22. APM

      1. Getting Started

      2. Visualizing Application Bottlenecks

      3. Using APM

        1. Filters

        2. Services overview

        3. Traces overview

        4. Transaction overview

        5. Span timeline

        6. Errors overview

        7. Metrics overview

        8. Machine Learning integration

        9. APM Agent configuration

        10. Advanced queries

    23. Uptime

      1. Overview

      2. Monitor

    24. SIEM

      1. Using the SIEM UI

      2. Anomaly Detection with Machine Learning

    25. Dev Tools

      1. Console

      2. Profiling queries and aggregations

        1. Getting Started

        2. Profiling a more complicated query

        3. Rendering pre-captured profiler JSON

      3. Debugging grok expressions

    26. Stack Monitoring

      1. Beats Metrics

      2. Cluster Alerts

      3. Elasticsearch Metrics

      4. Kibana Metrics

      5. Logstash Metrics

      6. Troubleshooting

    27. Management

      1. License Management

      2. Index patterns

        1. Cross-cluster search

      3. Rollup jobs

      4. Index lifecycle policies

        1. Creating an index lifecycle policy

        2. Managing index lifecycle policies

        3. Adding a policy to an index

        4. Example of using an index lifecycle policy

      5. Managing Fields

        1. String Field Formatters

        2. Date Field Formatters

        3. Geographic Point Field Formatters

        4. Numeric Field Formatters

        5. Scripted Fields

      6. Index management

      7. Setting advanced options

      8. Saved objects

      9. Managing Beats

      10. Working with remote clusters

      11. Snapshot and Restore

      12. Spaces

      13. Security

        1. Granting access to Kibana

        2. Kibana role management

        3. Kibana privileges

      14. Watcher

      15. Upgrade Assistant

    28. Reporting from Kibana

      1. Automating report generation

      2. PDF layout modes

      3. Reporting configuration

        1. Reporting and security

        2. Secure the reporting endpoints

        3. Chromium sandbox

      4. Troubleshooting

      5. Reporting integration

    29. REST API

      1. Features API

        1. Get features

      2. Kibana Spaces APIs

        1. Create space

        2. Update space

        3. Get space

        4. Get all spaces

        5. Delete space

        6. Copy saved objects to space

        7. Resolve copy to space conflicts

      3. Kibana role management APIs

        1. Create or update role

        2. Get specific role

        3. Get all roles

        4. Delete role

      4. Saved objects APIs

        1. Get object

        2. Bulk get objects

        3. Find objects

        4. Create object

        5. Bulk create objects

        6. Update object

        7. Delete object

        8. Export objects

        9. Import objects

        10. Resolve import errors

      5. Dashboard import and export APIs

        1. Import dashboard

        2. Dashboard export

      6. Logstash configuration management APIs

        1. Create pipeline

        2. Retrieve pipeline

        3. Delete pipeline

        4. List pipeline

      7. URL shortening API

        1. Shorten URL

      8. Upgrade assistant APIs

        1. Upgrade readiness status

        2. Start or resume reindex

        3. Check reindex status

        4. Cancel reindex

    30. Kibana plugins

      1. Install plugins

      2. Update and remove plugins

      3. Disable plugins

      4. Configure the plugin manager

      5. Known Plugins

    31. Limitations

      1. Nested Objects

      2. Exporting data

    32. Developer guide

      1. Core Development

        1. Considerations for basePath

        2. Managing Dependencies

        3. Modules and Autoloading

        4. Communicating with Elasticsearch

        5. Unit Testing

        6. Functional Testing

      2. Plugin Development

        1. Plugin Resources

        2. UI Exports

        3. Plugin feature registration

        4. Functional Tests for Plugins

        5. Localization for plugins

      3. Developing Visualizations

        1. Embedding Visualizations

        2. Developing Visualizations

        3. Visualization Factory

        4. Visualization Editors

        5. Visualization Request Handlers

        6. Visualization Response Handlers

        7. Vis object

        8. AggConfig object

      4. Add Data Guide

      5. Security

        1. Role-based access control

      6. Pull request review guidelines

      7. Interpreting CI Failures

  4. Beats Platform

    1. Community Beats

    2. Getting started with Beats

    3. Config file format

      1. Namespacing

      2. Config file data types

      3. Environment variables

      4. Reference variables

      5. Config file ownership and permissions

      6. Command line arguments

      7. YAML tips and gotchas

    4. Upgrading

      1. Upgrade between minor versions

      2. Upgrade from 6.x to 7.x

      3. Troubleshooting Beats upgrade issues

  5. Beats Developer Guide

    1. Contributing to Beats

    2. Community Beats

    3. Creating a New Beat

      1. Getting Ready

      2. Overview

      3. Generating Your Beat

      4. Fetching Dependencies and Setting up the Beat

      5. Building and Running the Beat

      6. The Beater Interface

      7. Sharing Your Beat with the Community

      8. Naming Conventions

    4. Creating New Kibana Dashboards

      1. Importing Existing Beat Dashboards

      2. Building Your Own Beat Dashboards

      3. Generating the Beat Index Pattern

      4. Exporting New and Modified Beat Dashboards

      5. Archiving Your Beat Dashboards

      6. Sharing Your Beat Dashboards

    5. Adding a New Protocol to Packetbeat

      1. Getting Ready

      2. Protocol Modules

      3. Testing

    6. Extending Metricbeat

      1. Overview

      2. Creating a Metricset

      3. Metricset Details

      4. Creating a Metricbeat Module

      5. Creating a Beat based on Metricbeat

      6. Metricbeat Developer FAQ

    7. Creating a New Filebeat Module

    8. Migrating dashboards from Kibana 5.x to 6.x

  6. Filebeat

    1. Overview

    2. Getting Started With Filebeat

      1. Step 1: Install Filebeat

      2. Step 2: Configure Filebeat

      3. Step 3: Load the index template in Elasticsearch

      4. Step 4: Set up the Kibana dashboards

      5. Step 5: Start Filebeat

      6. Step 6: View the sample Kibana dashboards

      7. Quick start: modules for common log formats

      8. Repositories for APT and YUM

    3. Setting up and running Filebeat

      1. Directory layout

      2. Secrets keystore

      3. Command reference

      4. Running Filebeat on Docker

      5. Running Filebeat on Kubernetes

      6. Filebeat and systemd

      7. Stopping Filebeat

    4. Upgrading Filebeat

    5. How Filebeat works

    6. Configuring Filebeat

      1. Specify which modules to run

      2. Configure inputs

      3. Manage multiline messages

      4. Specify general settings

      5. Load external configuration files

      6. Configure the internal queue

      7. Configure the output

      8. Configure index lifecycle management

      9. Load balance the output hosts

      10. Specify SSL settings

      11. Filter and enhance the exported data

      12. Parse data by using ingest node

      13. Enrich events with geoIP information

      14. Configure project paths

      15. Configure the Kibana endpoint

      16. Load the Kibana dashboards

      17. Load the Elasticsearch index template

      18. Configure logging

      19. Use environment variables in the configuration

      20. Autodiscover

      21. YAML tips and gotchas

      22. Regular expression support

      23. HTTP Endpoint

      24. filebeat.reference.yml

    7. Beats central management

      1. How central management works

      2. Enroll Beats in central management

    8. Modules

      1. Modules overview

      2. Apache module

      3. Auditd module

      4. AWS module

      5. CEF module

      6. Cisco module

      7. Coredns Module

      8. Elasticsearch module

      9. Envoyproxy Module

      10. Google Cloud module

      11. haproxy module

      12. IBM MQ module

      13. Icinga module

      14. IIS module

      15. Iptables module

      16. Kafka module

      17. Kibana module

      18. Logstash module

      19. MongoDB module

      20. MSSQL module

      21. MySQL module

      22. nats module

      23. NetFlow module

      24. Nginx module

      25. Osquery module

      26. Palo Alto Networks module

      27. PostgreSQL module

      28. RabbitMQ module

      29. Redis module

      30. Santa module

      31. Suricata module

      32. System module

      33. Traefik module

      34. Zeek (Bro) Module

    9. Exported fields

      1. Apache fields

      2. Auditd fields

      3. AWS fields

      4. Beat fields

      5. Decode CEF processor fields fields

      6. CEF fields

      7. Cisco fields

      8. Cloud provider metadata fields

      9. Coredns fields

      10. Docker fields

      11. ECS fields

      12. elasticsearch fields

      13. Envoyproxy fields

      14. Google Cloud fields

      15. haproxy fields

      16. Host fields

      17. ibmmq fields

      18. Icinga fields

      19. IIS fields

      20. iptables fields

      21. Jolokia Discovery autodiscover provider fields

      22. Kafka fields

      23. kibana fields

      24. Kubernetes fields

      25. Log file content fields

      26. logstash fields

      27. mongodb fields

      28. mssql fields

      29. MySQL fields

      30. nats fields

      31. NetFlow fields

      32. NetFlow fields

      33. Nginx fields

      34. Osquery fields

      35. panw fields

      36. PostgreSQL fields

      37. Process fields

      38. RabbitMQ fields

      39. Redis fields

      40. s3 fields

      41. Google Santa fields

      42. Suricata fields

      43. System fields

      44. Traefik fields

      45. Zeek fields

    10. Monitoring Filebeat

      1. Internal collection

        1. Settings for internal monitoring collection

      2. Metricbeat collection

    11. Securing Filebeat

      1. Secure communication with Elasticsearch

      2. Secure communication with Logstash

      3. Use X-Pack security

        1. Grant users access to secured resources

        2. Configure authentication credentials

        3. Configure Filebeat to use encrypted connections

      4. Use Linux Secure Computing Mode (seccomp)

    12. Troubleshooting

      1. Get help

      2. Debug

      3. Common problems

        1. Can’t read log files from network volumes

        2. Filebeat isn’t collecting lines from a file

        3. Too many open file handlers

        4. Registry file is too large

        5. Inode reuse causes Filebeat to skip lines

        6. Log rotation results in lost or duplicate events

        7. Open file handlers cause issues with Windows file rotation

        8. Filebeat is using too much CPU

        9. Dashboard in Kibana is breaking up data fields incorrectly

        10. Fields are not indexed or usable in Kibana visualizations

        11. Filebeat isn’t shipping the last line of a file

        12. Filebeat keeps open file handlers of deleted files for a long time

        13. Filebeat uses too much bandwidth

        14. Error loading config file

        15. Found unexpected or unknown characters

        16. Logstash connection doesn’t work

        17. @metadata is missing in Logstash

        18. Not sure whether to use Logstash or Beats

        19. SSL client fails to connect to Logstash

        20. Monitoring UI shows fewer Beats than expected

    13. A. Contributing to Beats

 

:

[Kibana] visualize 에서 scripted field 사용하기

Elastic/Kibana 2019. 4. 25. 14:14

예전에는 이 기능이 없어서 code 로 되어 있던 field 에 대해서 가독성 확보가 어려웠는데요.

나온지 한 참 되었는데 혹시 필요 하신 분들이 있을 수도 있어서 기록해 봅니다.

 

사실은 제가 기억력이 좋지 않아서 복습 차원에 정리 해 보는 내용입니다.

 

공식 문서)

https://www.elastic.co/guide/en/kibana/current/scripted-fields.html

 

Scripted Fields | Kibana User Guide [7.0] | Elastic

Use of Groovy, JavaScript, and Python scripting is deprecated starting in Elasticsearch 5.0, and support for those scripting languages will be removed in the future.

www.elastic.co

보통 visualize 에서도 "Advanced" 클릭 하시면 JSON Input 영역에 필요한 script 를 넣을 수 있습니다.

Terms aggregation 을 기본 예로 들면,

{
  "aggs": {
    "1": {
      "terms": {
        "field": "user",
        "size": 100000,
        "order": {
          "_count": "desc"
        }
      }
...중략...
}

Kibana 에서 이 정도 작성이 가능 합니다.

여기서 "min_doc_count" 같은 값을 설정 하고 싶을 경우 위에 이야기한 "Advanced" 를 클릭해서 값을 넣으시면 아래와 같이 등록이 됩니다.

 

Advacned JSON Input)

{
  "min_doc_count":3
}

 

최종 Request string)

{
  "aggs": {
    "1": {
      "terms": {
        "field": "user",
        "size": 100000,
        "order": {
          "_count": "desc"
        },
        "min_doc_count":3
      }
...중략...
}

그럼 scripted field 추가를 해보겠습니다.

공식 문서에 자세히 나와 있기 때문에 별도 화면 캡쳐 등의 설명은 하지 않겠습니다.

내용만 발췌 합니다.

Creating a Scripted Field

To create a scripted field:

  1. Go to Management > Kibana > Index Patterns

  2. Select the index pattern you want to add a scripted field to.

  3. Go to the pattern’s Scripted fields tab.

  4. Click Add scripted field.

  5. Enter a name for the scripted field.

  6. Enter the expression that you want to use to compute a value on the fly from your index data.

  7. Click Create field.

이렇게 생성 완료 후 Visualize 에서 Terms aggs 작성 하시면서 Field 쪽에 작성 하신 scripted field 를 넣어 주시면 됩니다.

단, 생성할 때 선택한 index 로 visualize 를 만드셔야지 scripted field 가 노출 됩니다.

 

아래는 제가 사용한 scripted field 정보 입니다.

Name : code_ko
Lang : painless
Script :
if (doc['code'].value == '1') { '카테고리1' } 
else if (doc['code'].value == '2') {'카테고리2'} 
else if (doc['code'].value == '3') {'카테고리3'} 
else if (doc['code'].value == '4') {'카테고리4'} 
else if (doc['code'].value == '5') {'카테고리5'} 
else if (doc['code'].value == '6') {'카테고리6'} 
else if (doc['code'].value == '7') {'카테고리7'} 
else if (doc['code'].value == '8') {'카테고리8'} 
else if (doc['code'].value == '9') {'카테고리9'} 
else if (doc['code'].value == '10') {'카테고리10'} 
else if (doc['code'].value == '11') {'카테고리11'} 
else if (doc['code'].value == '12') {'카테고리12'}

- pseudocode 로 변경해서 기록 했습니다.

:

[Kibana] start and stop 스크립트

Elastic/Kibana 2018. 7. 5. 09:52

Kibana 실행 스크립트.

$ bin/kibana 귀찮아서 그냥 es 에서 쓰던거 복붙.


[start.sh]


#!/bin/bash


DIR_NAME=`dirname "$0"`

DIR_HOME=`cd $DIR_NAME; cd ..; pwd`


$DIR_HOME/bin/kibana> /dev/null 2>&1 & KIBANA_PID=$!

echo $KIBANA_PID > $DIR_HOME/bin/kibana.pid



[stop.sh]


#!/bin/bash


DIR_NAME=`dirname "$0"`

DIR_HOME=`cd $DIR_NAME; cd ..; pwd`


KIBANA_PID=`cat $DIR_HOME/bin/kibana.pid`

kill -9 $KIBANA_PID

echo "Kibana Daemon($KIBANA_PID) is killed"

rm -f $DIR_HOME/bin/kibana.pid


: