'logstash'에 해당되는 글 38건

  1. 2019.11.06 [Logstash] logstash filter date 조금 알아보기
  2. 2019.11.05 [Elastic] 목차 입니다.
  3. 2019.11.04 [Logstash] JSON filter plugin
  4. 2019.10.17 [Elasticsearch] 앱 내 사용자 행동로그 수집 파이프라인 구성
  5. 2018.10.04 [Logstash] --config.reload.automatic 사용 경험 공유
  6. 2018.08.28 [Logstash] AWS SQS + Logstash 구성
  7. 2018.07.25 [Beats] 오랜만에 Filebeat 설치 1
  8. 2018.07.09 [Logstash] http output plugin - slack chat
  9. 2018.04.24 [Logstash] Logstash 를 이용한 CSV 파일 Import를 하려면
  10. 2017.08.17 [Logstash] input file start_position => "end"

[Logstash] logstash filter date 조금 알아보기

Elastic/Logstash 2019. 11. 6. 14:34

문의가 들어 왔습니다.

여러 필드에 대해서 date format 이 다른데 어떻게 적용을 해야 하나요?

 

그래서 소스코드를 열어 보고 아래와 같이 해보라고 했습니다.

date {
...
}

date {
...
}

결국 date {...} 를 필드 별로 선언을 해주면 되는 내용입니다.

 

공식 문서)

https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

 

Common Options)

https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-common-options

 

Date Filter Configuration Options)

Setting

Input type

Required

locale

string

No

match

array

No

tag_on_failure

array

No

target

string

No

timezone

string

No

전체 옵션이 필수가 아니긴 합니다.

그래도 꼭 아셔야 하는 설정은 match, target 입니다.

 

- match 의 첫 번째 값은 field 명이고, 그 이후는 format 들이 되겠습니다.

(공식 문서에 잘 나와 있습니다.)

 

An array with field name first, and format patterns following, [ field, formats... ]

If your time field has multiple possible formats, you can do this:

 

match => [ "logdate", 

    "MMM dd yyyy HH:mm:ss", 

    "MMM d yyyy HH:mm:ss", 

    "ISO8601" ]

 

- target 은 지정을 하지 않게 되면 기본 @timestamp 필드로 설정이 됩니다. 변경 하고자 하면 target 에 원하시는 field name 을 넣으시면 됩니다.

 

예제)

date {
    match => ["time" , "yyyy-MM-dd'T'HH:mm:ssZ", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
    target => "@timestamp"
}

date {
    match => ["localtime" , "yyyy-MM-dd HH:mm:ssZ"]
    target => "time"
}

DateFilter.java)

더보기
/*
 * Licensed to Elasticsearch under one or more contributor
 * license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright
 * ownership. Elasticsearch licenses this file to you under
 * the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

package org.logstash.filters;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.joda.time.Instant;
import org.logstash.Event;
import org.logstash.ext.JrubyEventExtLibrary.RubyEvent;
import org.logstash.filters.parser.CasualISO8601Parser;
import org.logstash.filters.parser.JodaParser;
import org.logstash.filters.parser.TimestampParser;
import org.logstash.filters.parser.TimestampParserFactory;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public class DateFilter {
  private static Logger logger = LogManager.getLogger();
  private final String sourceField;
  private final String[] tagOnFailure;
  private RubyResultHandler successHandler;
  private RubyResultHandler failureHandler;
  private final List<ParserExecutor> executors = new ArrayList<>();
  private final ResultSetter setter;

  public interface RubyResultHandler {
    void handle(RubyEvent event);
  }

  public DateFilter(String sourceField, String targetField, List<String> tagOnFailure, RubyResultHandler successHandler, RubyResultHandler failureHandler) {
    this(sourceField, targetField, tagOnFailure);
    this.successHandler = successHandler;
    this.failureHandler = failureHandler;
  }

  public DateFilter(String sourceField, String targetField, List<String> tagOnFailure) {
    this.sourceField = sourceField;
    this.tagOnFailure = tagOnFailure.toArray(new String[0]);
    if (targetField.equals("@timestamp")) {
      this.setter = new TimestampSetter();
    } else {
      this.setter = new FieldSetter(targetField);
    }
  }

  public void acceptFilterConfig(String format, String locale, String timezone) {
    TimestampParser parser = TimestampParserFactory.makeParser(format, locale, timezone);
    logger.debug("Date filter with format={}, locale={}, timezone={} built as {}", format, locale, timezone, parser.getClass().getName());
    if (parser instanceof JodaParser || parser instanceof CasualISO8601Parser) {
      executors.add(new TextParserExecutor(parser, timezone));
    } else {
      executors.add(new NumericParserExecutor(parser));
    }
  }

 public List<RubyEvent> receive(List<RubyEvent> rubyEvents) {
    for (RubyEvent rubyEvent : rubyEvents) {
      Event event = rubyEvent.getEvent();

      switch (executeParsers(event)) {
        case FIELD_VALUE_IS_NULL_OR_FIELD_NOT_PRESENT:
        case IGNORED:
          continue;
        case SUCCESS:
          if (successHandler != null) {
            successHandler.handle(rubyEvent);
          }
          break;
        case FAIL: // fall through
        default:
          for (String t : tagOnFailure) {
            event.tag(t);
          }
          if (failureHandler != null) {
            failureHandler.handle(rubyEvent);
          }
      }
    }
    return rubyEvents;
  }

  public ParseExecutionResult executeParsers(Event event) {
    Object input = event.getField(sourceField);
    if (event.isCancelled()) { return ParseExecutionResult.IGNORED; }
    if (input == null) { return ParseExecutionResult.FIELD_VALUE_IS_NULL_OR_FIELD_NOT_PRESENT; }

    for (ParserExecutor executor : executors) {
      try {
        Instant instant = executor.execute(input, event);
        setter.set(event, instant);
        return ParseExecutionResult.SUCCESS;
      } catch (IllegalArgumentException | IOException e) {
        // do nothing, try next ParserExecutor
      }
    }
    return ParseExecutionResult.FAIL;
  }
}

 

:

[Elastic] 목차 입니다.

Elastic 2019. 11. 5. 14:52

Elastic Stack 의 Reference 목차 입니다.

이걸 왜 한 장으로 정리를 했냐면 목차만 잘 찾아 봐도 해결 방법이 어딨는지 어떤 기능을 제공 하고 있는지 쉽게 알수 있습니다.

(In my case!!)

 

그래서 혼자 보기 아까워서 그냥 올려봤습니다.

Elastic Stack References)

1. Elasticsearch

2. Logstash

3. Kibana

4. Beats Platform

5. Beats Developer Guide

6. Filebeat

더보기
  1. Elasticsearch

    1. Elasticsearch introduction

      1. Data in: documents and indices

      2. Information out: search and analyze

      3. Scalability and resilience

    2. Getting Started with Elasticsearch

      1. Get Elasticsearch up and running

      2. Index some documents

      3. Start searching

      4. Analyze results with aggregations

      5. Where to go from here

    3. Set up Elasticsearch

      1. Installing Elasticsearch

        1. Install Elasticsearch from archive on Linux or MacOS

        2. Install Elasticsearch with .zip on Windows

        3. Install Elasticsearch with Debian Package

        4. Install Elasticsearch with RPM

        5. Install Elasticsearch Windows MSI Installer

        6. Install Elasticsearch with Docker

        7. Install Elasticsearch on macOS with Homebrew

      2. Configuring Elasticsearch

        1. Setting JVM options

        2. Secure settings

        3. Logging configuration

        4. Auditing settings

        5. Cross-cluster replication settings

        6. Transforms settings

        7. Index lifecycle management settings

        8. License settings

        9. Machine learning settings

        10. Security settings

        11. SQL access settings

        12. Watcher settings

      3. Important Elasticsearch configuration

        1. path.data and path.logs

        2. cluster.name

        3. node.name

        4. network.host

        5. Discovery and cluster formation settings

        6. Setting the heap size

        7. JVM heap dump path

        8. GC logging

        9. Temp directory

        10. JVM fatal error logs

      4. Important System Configuration

        1. Configuring system settings

        2. Disable swapping

        3. File Descriptors

        4. Virtual memory

        5. Number of threads

        6. DNS cache settings

        7. JNA temporary directory not mounted with noexec

      5. Bootstrap Checks

        1. Heap size check

        2. File descriptor check

        3. Memory lock check

        4. Maximum number of threads check

        5. Max file size check

        6. Maximum size virtual memory check

        7. Maximum map count check

        8. Client JVM check

        9. Use serial collector check

        10. System call filter check

        11. OnError and OnOutOfMemoryError checks

        12. Early-access check

        13. G1GC check

        14. All permission check

        15. Discovery configuration check

      6. Starting Elasticsearch

      7. Stopping Elasticsearch

      8. Adding nodes to your cluster

      9. Set up X-Pack

      10. Configuring X-Pack Java Clients

      11. Bootstrap Checks for X-Pack

    4. Upgrade Elasticsearch

      1. Rolling upgrades

      2. Full cluster restart upgrade

      3. Reindex before upgrading

        1. Reindex in place

        2. Reindex from a remote cluster

    5. Aggregations

      1. Metrics Aggregations

        1. Avg Aggregation

        2. Weighted Avg Aggregation

        3. Cardinality Aggregation

        4. Extended Stats Aggregation

        5. Geo Bounds Aggregation

        6. Geo Centroid Aggregation

        7. Max Aggregation

        8. Min Aggregation

        9. Percentiles Aggregation

        10. Percentile Ranks Aggregation

        11. Scripted Metric Aggregation

        12. Stats Aggregation

        13. Sum Aggregation

        14. Top Hits Aggregation

        15. Value Count Aggregation

        16. Median Absolute Deviation Aggregation

      2. Bucket Aggregations

        1. Adjacency Matrix Aggregation

        2. Auto-interval Date Histogram Aggregation

        3. Children Aggregation

        4. Composite Aggregation

        5. Date Histogram Aggregation

        6. Date Range Aggregation

        7. Diversified Sampler Aggregation

        8. Filter Aggregation

        9. Filters Aggregation

        10. Geo Distance Aggregation

        11. GeoHash grid Aggregation

        12. GeoTile Grid Aggregation

        13. Global Aggregation

        14. Histogram Aggregation

        15. IP Range Aggregation

        16. Missing Aggregation

        17. Parent Aggregation

        18. Range Aggregation

        19. Rare Terms Aggregation

        20. Reverse nested Aggregation

        21. Sampler Aggregation

        22. Significant Terms Aggregation

        23. Significant Text Aggregation

        24. Terms Aggregation

        25. Subtleties of bucketing range fields

      3. Pipeline Aggregations

        1. Avg Bucket Aggregation

        2. Derivative Aggregation

        3. Max Bucket Aggregation

        4. Min Bucket Aggregation

        5. Sum Bucket Aggregation

        6. Stats Bucket Aggregation

        7. Extended Stats Bucket Aggregation

        8. Percentiles Bucket Aggregation

        9. Moving Average Aggregation

        10. Moving Function Aggregation

        11. Cumulative Sum Aggregation

        12. Cumulative Cardinality Aggregation

        13. Bucket Script Aggregation

        14. Bucket Selector Aggregation

        15. Bucket Sort Aggregation

        16. Serial Differencing Aggregation

      4. Matrix Aggregations

        1. Matrix Stats

      5. Caching heavy aggregations

      6. Returning only aggregations

      7. Aggregation Metadata

      8. Returning the type of the aggregation

    6. Query DSL

      1. Query and filter context

      2. Compound queries

        1. Boolean

        2. Boosting

        3. Constant score

        4. Disjunction score

        5. Function score

      3. Full text queries

        1. Intervals

        2. Match

        3. Match boolean prefix

        4. Match phrase

        5. Match phrase prefix

        6. Multi-match

        7. Common Terms Query

        8. Query String

        9. Simple query string

      4. Geo queries

        1. Geo-bounding box

        2. Geo-distance

        3. Geo-polygon

        4. Geo-shape

      5. Shape queries

        1. Shape

      6. Joining queries

        1. Nested

        2. Has child

        3. Has parent

        4. Parent ID

      7. Match all

      8. Span queries

        1. Span containing

        2. Span field masking

        3. Span first

        4. Span multi-term

        5. Span near

        6. Span not

        7. Span or

        8. Span term

        9. Span within

      9. Specialized queries

        1. Distance feature

        2. More like this

        3. Percolate

        4. Rank feature

        5. Script

        6. Script score

        7. Wrapper

        8. Pinned Query

      10. Term-level queries

        1. Exists

        2. Fuzzy

        3. IDs

        4. Prefix

        5. Range

        6. Regexp

        7. Term

        8. Terms

        9. Terms set

        10. Type Query

        11. Wildcard

      11. minimum_should_match parameter

      12. rewrite parameter

      13. Regular expression syntax

    7. Search across clusters

    8. Scripting

      1. How to use scripts

      2. Accessing document fields and special variables

      3. Scripting and security

      4. Painless scripting language

      5. Lucene expressions language

      6. Advanced scripts using script engines

    9. Mapping

      1. Removal of mapping types

      2. Field datatypes

        1. Alias

        2. Arrays

        3. Binary

        4. Boolean

        5. Date

        6. Date nanoseconds

        7. Dense vector

        8. Flattened

        9. Geo-point

        10. Geo-shape

        11. IP

        12. Join

        13. Keyword

        14. Nested

        15. Numeric

        16. Object

        17. Percolator

        18. Range

        19. Rank feature

        20. Rank features

        21. Search-as-you-type

        22. Sparse vector

        23. Text

        24. Token count

        25. Shape

      3. Meta-Fields

        1. _field_names field

        2. _ignored field

        3. _id field

        4. _index field

        5. _meta field

        6. _routing field

        7. _source field

        8. _type field

      4. Mapping parameters

        1. analyzer

        2. normalizer

        3. boost

        4. coerce

        5. copy_to

        6. doc_values

        7. dynamic

        8. enabled

        9. eager_global_ordinals

        10. fielddata

        11. format

        12. ignore_above

        13. ignore_malformed

        14. index

        15. index_options

        16. index_phrases

        17. index_prefixes

        18. fields

        19. norms

        20. null_value

        21. position_increment_gap

        22. properties

        23. search_analyzer

        24. similarity

        25. store

        26. term_vector

      5. Dynamic Mapping

        1. Dynamic field mapping

        2. Dynamic templates

    10. Analysis

      1. Anatomy of an analyzer

      2. Testing analyzers

      3. Analyzers

        1. Configuring built-in analyzers

        2. Fingerprint Analyzer

        3. Keyword Analyzer

        4. Language Analyzers

        5. Pattern Analyzer

        6. Simple Analyzer

        7. Standard Analyzer

        8. Stop Analyzer

        9. Whitespace Analyzer

        10. Custom Analyzer

      4. Normalizers

      5. Tokenizers

        1. Char Group Tokenizer

        2. Classic Tokenizer

        3. Edge NGram Tokenizer

        4. Keyword Tokenizer

        5. Letter Tokenizer

        6. Lowercase Tokenizer

        7. NGram Tokenizer

        8. Path Hierachy Tokenizer

        9. Pattern Tokenizer

        10. Simple Pattern Tokenizer

        11. Simple Pattern Split Tokenizer

        12. Standard Tokenizer

        13. Thai Tokenizer

        14. UAX URL Email Tokenizer

        15. Whitespace Tokenizer

      6. Token Filters

        1. Apostrophe

        2. ASCII Folding Token Filter

        3. CJK bigram

        4. CJK width

        5. Classic Token Filter

        6. Common Grams Token Filter

        7. Compound Word Token Filters

        8. Conditional Token Filter

        9. Decimal Digit Token Filter

        10. Delimited Payload Token Filter

        11. Edge NGram Token Filter

        12. Elision Token Filter

        13. Fingerprint Token Filter

        14. Flatten Graph Token Filter

        15. Hunspell Token Filter

        16. Keep Types Token Filter

        17. Keep Words Token Filter

        18. Keyword Request Token Filter

        19. KStem Token Filter

        20. Length Token Filter

        21. Limit Token Count Token Filter

        22. Lowercase Token Filter

        23. MinHash Token Filter

        24. Multiplexer Token Filter

        25. NGram Token Filter

        26. Normalization Token Filter

        27. Pattern Capture Token Filter

        28. Pattern Replace Token Filter

        29. Phonetic Token Filter

        30. Porter Stem Token Filter

        31. Predicate Token Filter Script

        32. Remove Duplicates Token Filter

        33. Reverse Token Filter

        34. Shingle Token Filter

        35. Snowball Token Filter

        36. Stemmer Token Filter

        37. Stemmer Override Token Filter

        38. Stop Token Filter

        39. Synonym Token Filter

        40. Synonym Graph Token Filter

        41. Trim Token Filter

        42. Truncate Token Filter

        43. Unique Token Filter

        44. Uppercase Token Filter

        45. Word Delimiter Token Filter

        46. Word Delimiter Graph Token Filter

      7. Character Filters

        1. HTML Strip Char Filter

        2. Mapping Char Filter

        3. Pattern Replace Char Filter

    11. Modules

      1. Discovery and cluster formation

        1. Discovery

        2. Quorum-based decision making

        3. Voting configurations

        4. Bootstrapping a cluster

        5. Adding and removing nodes

        6. Publishing the cluster state

        7. Cluster fault detection

        8. Discovery and cluster formation settings

      2. Shard allocation and cluster-level routing

        1. Cluster level shard allocation

        2. Disk-based shard allocation

        3. Shard allocation awareness

        4. Cluster-level shard allocation filtering

        5. Miscellaneous cluster settings

      3. Local Gateway

        1. Dangling indices

      4. HTTP

      5. Indices

        1. Circuit Breaker

        2. Fielddata

        3. Node Query Cache

        4. Indexing Buffer

        5. Shard request cache

        6. Index recovery

        7. Search Settings

      6. Network Settings

      7. Node

      8. Plugins

      9. Snapshot And Restore

      10. Thread Pool

      11. Transport

      12. Remote clusters

    12. Index modules

      1. Analysis

      2. Index Shard Allocation

        1. Index-level shard allocation filtering

        2. Delaying allocation when a node leaves

        3. Index recovery prioritization

        4. Total shards per node

      3. Mapper

      4. Merge

      5. Similarity module

      6. Slow Log

      7. Store

        1. Preloading data into the file system cache

      8. Translog

      9. History retention

      10. Index Sorting

        1. Use index sorting to speed up conjunctions

    13. Ingest node

      1. Pipeline Definition

      2. Accessing Data in Pipelines

      3. Conditional Execution in Pipelines

        1. Handling Nested Fields in Conditionals

        2. Complex Conditionals

        3. Conditionals with the Pipeline Processor

        4. Conditionals with the Regular Expressions

      4. Handling Failures in Pipelines

      5. Processors

        1. Append Processor

        2. Bytes Processor

        3. Circle Processor

        4. Convert Processor

        5. Date Processor

        6. Date Index Name Processor

        7. Dissect Processor

        8. Dot Expander Processor

        9. Drop Processor

        10. Fail Processor

        11. Foreach Processor

        12. GeoIP Processor

        13. Grok Processor

        14. Gsub Processor

        15. HTML Strip Processor

        16. Join Processor

        17. JSON Processor

        18. KV Processor

        19. Lowercase Processor

        20. Pipeline Processor

        21. Remove Processor

        22. Rename Processor

        23. Script Processor

        24. Set Processor

        25. Set Security User Processor

        26. Split Processor

        27. Sort Processor

        28. Trim Processor

        29. Uppercase Processor

        30. URL Decode Processor

        31. User Agent processor

    14. Managing the index lifecycle

      1. Getting started with index lifecycle management

      2. Policy phases and actions

        1. Timing

        2. Phase Execution

        3. Actions

        4. Full Policy

      3. Set up index lifecycle management policy

        1. Applying a policy to an index template

        2. Apply a policy to a create index request

      4. Using policies to manage index rollover

        1. Skipping Rollover

      5. Update policy

        1. Updates to policies not managing indices

        2. Updates to executing policies

        3. Switching policies for an index

      6. Index lifecycle error handling

      7. Restoring snapshots of managed indices

      8. Start and stop index lifecycle management

      9. Using ILM with existing indices

        1. Managing existing periodic indices with ILM

        2. Reindexing via ILM

      10. Getting started with snapshot lifecycle management

    15. SQL access

      1. Overview

      2. Getting Started with SQL

      3. Conventions and Terminology

        1. Mapping concepts across SQL and Elasticsearch

      4. Security

      5. SQL REST API

        1. Overview

        2. Response Data Formats

        3. Paginating through a large response

        4. Filtering using Elasticsearch query DSL

        5. Columnar results

        6. Supported REST parameters

      6. SQL Translate API

      7. SQL CLI

      8. SQL JDBC

        1. API usage

      9. SQL ODBC

        1. Driver installation

        2. Configuration

      10. SQL Client Applications

        1. DBeaver

        2. DbVisualizer

        3. Microsoft Excel

        4. Microsoft Power BI Desktop

        5. Microsoft PowerShell

        6. MicroStrategy Desktop

        7. Qlik Sense Desktop

        8. SQuirreL SQL

        9. SQL Workbench/J

        10. Tableau Desktop

      11. SQL Language

        1. Lexical Structure

        2. SQL Commands

        3. DESCRIBE TABLE

        4. SELECT

        5. SHOW COLUMNS

        6. SHOW FUNCTIONS

        7. SHOW TABLES

        8. Data Types

        9. Index patterns

        10. Frozen Indices

      12. Functions and Operators

        1. Comparison Operators

        2. Logical Operators

        3. Math Operators

        4. Cast Operators

        5. LIKE and RLIKE Operators

        6. Aggregate Functions

        7. Grouping Functions

        8. Date/Time and Interval Functions and Operators

        9. Full-Text Search Functions

        10. Mathematical Functions

        11. String Functions

        12. Type Conversion Functions

        13. Geo Functions

        14. Conditional Functions And Expressions

        15. System Functions

      13. Reserved keywords

      14. SQL Limitations

    16. Monitor a cluster

      1. Overview

      2. How it works

      3. Monitoring in a production environment

      4. Collecting monitoring data

        1. Pausing data collection

      5. Collecting monitoring data with Metricbeat

      6. Collecting log data with Filebeat

      7. Configuring indices for monitoring

      8. Collectors

      9. Exporters

        1. Local exporters

        2. HTTP exporters

      10. Troubleshooting

    17. Frozen indices

      1. Best practices

      2. Searching a frozen index

      3. Monitoring frozen indices

    18. Roll up or transform your data

      1. Rolling up historical data

        1. Overview

        2. API quick reference

        3. Getting started

        4. Understanding groups

        5. Rollup aggregation limitations

        6. Rollup search limitations

      2. Transforming data

        1. Overview

        2. When to use transforms

        3. How checkpoints work

        4. API quick reference

        5. Tutorial: Transforming the eCommerce sample data

        6. Examples

        7. Troubleshooting

        8. Limitations

    19. Set up a cluster for high availability

      1. Back up a cluster

        1. Back up the data

        2. Back up the cluster configuration

        3. Back up the security configuration

        4. Restore the security configuration

        5. Restore the data

      2. Cross-cluster replication

        1. Overview

        2. Requirements for leader indices

        3. Automatically following indices

        4. Getting started with cross-cluster replication

        5. Remote recovery

        6. Upgrading clusters

    20. Secure a cluster

      1. Overview

      2. Configuring security

        1. Encrypting communications in Elasticsearch

        2. Encrypting communications in an Elasticsearch Docker Container

        3. Enabling cipher suites for stronger encryption

        4. Separating node-to-node and client traffic

        5. Configuring an Active Directory realm

        6. Configuring a file realm

        7. Configuring an LDAP realm

        8. Configuring a native realm

        9. Configuring a PKI realm

        10. Configuring a SAML realm

        11. Configuring a Kerberos realm

        12. Security files

        13. FIPS 140-2

      3. How security works

      4. User authentication

        1. Built-in users

        2. Internal users

        3. Realms

        4. Realm chains

        5. Active Directory user authentication

        6. File-based user authentication

        7. LDAP user authentication

        8. Native user authentication

        9. PKI user authentication

        10. SAML authentication

        11. Kerberos authentication

        12. Integrating with other authentication systems

        13. Enabling anonymous access

        14. Controlling the user cache

      5. Configuring SAML single-sign-on on the Elastic Stack

        1. The identity provider

        2. Configure Elasticsearch for SAML authentication

        3. Generating SP metadata

        4. Configuring role mappings

        5. User metadata

        6. Configuring Kibana

        7. Troubleshooting SAML Realm Configuration

      6. Configuring single sign-on to the Elastic Stack using OpenID Connect

        1. The OpenID Connect Provider

        2. Configure Elasticsearch for OpenID Connect authentication

        3. Configuring role mappings

        4. User metadata

        5. Configuring Kibana

        6. OpenID Connect without Kibana

      7. User authorization

        1. Built-in roles

        2. Defining roles

        3. Security privileges

        4. Document level security

        5. Field level security

        6. Granting privileges for indices and aliases

        7. Mapping users and groups to roles

        8. Setting up field and document level security

        9. Submitting requests on behalf of other users

        10. Configuring authorization delegation

        11. Customizing roles and authorization

      8. Auditing security events

        1. Audit event types

        2. Logfile audit output

        3. Auditing search queries

      9. Encrypting communications

        1. Setting up TLS on a cluster

      10. Restricting connections with IP filtering

      11. Cross cluster search, clients, and integrations

        1. Cross cluster search and security

        2. Java Client and security

        3. HTTP/REST clients and security

        4. ES-Hadoop and Security

        5. Beats and Security

        6. Monitoring and security

      12. Tutorial: Getting started with security

        1. Enable Elasticsearch security features

        2. Create passwords for built-in users

        3. Add the built-in user to Kibana

        4. Configure authentication

        5. Create users

        6. Assign roles

        7. Add user information in Logstash

        8. View system metrics in Kibana

      13. Tutorial: Encrypting communications

        1. Generate certificates

        2. Encrypt internode communications

        3. Add nodes to your cluster

      14. Troubleshooting

        1. Some settings are not returned via the nodes settings API

        2. Authorization exceptions

        3. Users command fails due to extra arguments

        4. Users are frequently locked out of Active Directory

        5. Certificate verification fails for curl on Mac

        6. SSLHandshakeException causes connections to fail

        7. Common SSL/TLS exceptions

        8. Common Kerberos exceptions

        9. Common SAML issues

        10. Internal Server Error in Kibana

        11. Setup-passwords command fails due to connection failure

        12. Failures due to relocation of the configuration files

      15. Limitations

    21. Alerting on cluster and index events

      1. Getting started with Watcher

      2. How Watcher works

      3. Encrypting sensitive data in Watcher

      4. Inputs

        1. Simple input

        2. Search input

        3. HTTP input

        4. Chain input

      5. Triggers

        1. Schedule trigger

      6. Conditions

        1. Always condition

        2. Never condition

        3. Compare condition

        4. Array compare condition

        5. Script condition

      7. Actions

        1. Running an action for each element in an array

        2. Adding conditions to actions

        3. Email action

        4. Webhook action

        5. Index action

        6. Logging Action

        7. Slack Action

        8. PagerDuty action

        9. Jira action

      8. Transforms

        1. Search transform

        2. Script transform

        3. Chain transform

      9. Java API

      10. Managing watches

      11. Example watches

        1. Watching the status of an Elasticsearch cluster

        2. Watching event data

      12. Troubleshooting

      13. Limitations

    22. Command line tools

      1. elasticsearch-certgen

      2. elasticsearch-certutil

      3. elasticsearch-croneval

      4. elasticsearch-migrate

      5. elasticsearch-node

      6. elasticsearch-saml-metadata

      7. elasticsearch-setup-passwords

      8. elasticsearch-shard

      9. elasticsearch-syskeygen

      10. elasticsearch-users

    23. How To

      1. General recommendations

      2. Recipes

        1. Mixing exact search with stemming

        2. Getting consistent scoring

        3. Incorporating static relevance signals into the score

      3. Tune for indexing speed

      4. Tune for search speed

        1. Tune your queries with the Profile API

        2. Faster phrase queries with index_phrases

        3. Faster prefix queries with index_prefixes

      5. Tune for disk usage

    24. Testing

      1. Java Testing Framework

        1. Why randomized testing?

        2. Using the Elasticsearch test classes

        3. Unit tests

        4. Integration tests

        5. Randomized testing

        6. Assertions

    25. Glossary of terms

    26. REST APIs

      1. API conventions

        1. Multiple Indices

        2. Date math support in index names

        3. Common options

        4. URL-based access control

      2. cat APIs

        1. cat aliases

        2. cat allocation

        3. cat count

        4. cat fielddata

        5. cat health

        6. cat indices

        7. cat master

        8. cat nodeattrs

        9. cat nodes

        10. cat pending tasks

        11. cat plugins

        12. cat recovery

        13. cat repositories

        14. cat task management

        15. cat thread pool

        16. cat shards

        17. cat segments

        18. cat snapshots

        19. cat templates

      3. Cluster APIs

        1. Cluster Health

        2. Cluster State

        3. Cluster Stats

        4. Pending cluster tasks

        5. Cluster Reroute

        6. Cluster Update Settings

        7. Cluster Get Settings

        8. Nodes Stats

        9. Nodes Info

        10. Nodes Feature Usage

        11. Remote Cluster Info

        12. Task management

        13. Nodes hot_threads

        14. Cluster Allocation Explain API

        15. Voting Configuration Exclusions

      4. Cross-cluster replication APIs

        1. Get CCR stats

        2. Create follower

        3. Pause follower

        4. Resume follower

        5. Unfollow

        6. Forget follower

        7. Get follower stats

        8. Get follower info

        9. Create auto-follow pattern

        10. Delete auto-follow pattern

        11. Get auto-follow pattern

      5. Document APIs

        1. Reading and Writing documents

        2. Index

        3. Get

        4. Delete

        5. Delete by query

        6. Update

        7. Update By Query API

        8. Multi get

        9. Bulk

        10. Reindex

        11. Term vectors

        12. Multi term vectors

        13. ?refresh

        14. Optimistic concurrency control

      6. Explore API

      7. Index APIs

        1. Add index alias

        2. Analyze

        3. Clear cache

        4. Clone index

        5. Close index

        6. Create index

        7. Delete index

        8. Delete index alias

        9. Delete index template

        10. Flush

        11. Force merge

        12. Freeze index

        13. Get field mapping

        14. Get index

        15. Get index alias

        16. Get index settings

        17. Get index template

        18. Get mapping

        19. Index alias exists

        20. Index exists

        21. Index recovery

        22. Index segments

        23. Index shard stores

        24. Index stats

        25. Index template exists

        26. Open index

        27. Put index template

        28. Put mapping

        29. Refresh

        30. Rollover index

        31. Shrink index

        32. Split index

        33. Synced flush

        34. Type exists

        35. Unfreeze index

        36. Update index alias

        37. Update index settings

      8. Index lifecycle management API

        1. Create policy

        2. Get policy

        3. Delete policy

        4. Move to step

        5. Remove policy

        6. Retry policy

        7. Get index lifecycle management status

        8. Explain lifecycle

        9. Start index lifecycle management

        10. Stop index lifecycle management

      9. Ingest APIs

        1. Put pipeline

        2. Get pipeline

        3. Delete pipeline

        4. Simulate pipeline

      10. Info API

      11. Licensing APIs

        1. Delete license

        2. Get license

        3. Get trial status

        4. Start trial

        5. Get basic status

        6. Start basic

        7. Update license

      12. Machine learning anomaly detection APIs

        1. Add events to calendar

        2. Add jobs to calendar

        3. Close jobs

        4. Create jobs

        5. Create calendar

        6. Create datafeeds

        7. Create filter

        8. Delete calendar

        9. Delete datafeeds

        10. Delete events from calendar

        11. Delete filter

        12. Delete forecast

        13. Delete jobs

        14. Delete jobs from calendar

        15. Delete model snapshots

        16. Delete expired data

        17. Find file structure

        18. Flush jobs

        19. Forecast jobs

        20. Get buckets

        21. Get calendars

        22. Get categories

        23. Get datafeeds

        24. Get datafeed statistics

        25. Get influencers

        26. Get jobs

        27. Get job statistics

        28. Get machine learning info

        29. Get model snapshots

        30. Get overall buckets

        31. Get scheduled events

        32. Get filters

        33. Get records

        34. Open jobs

        35. Post data to jobs

        36. Preview datafeeds

        37. Revert model snapshots

        38. Set upgrade mode

        39. Start datafeeds

        40. Stop datafeeds

        41. Update datafeeds

        42. Update filter

        43. Update jobs

        44. Update model snapshots

      13. Machine learning data frame analytics APIs

        1. Create data frame analytics jobs

        2. Delete data frame analytics jobs

        3. Evaluate data frame analytics

        4. Estimate memory usage for data frame analytics jobs

        5. Get data frame analytics jobs

        6. Get data frame analytics jobs stats

        7. Start data frame analytics jobs

        8. Stop data frame analytics jobs

      14. Migration APIs

        1. Deprecation info

      15. Reload search analyzers

      16. Rollup APIs

        1. Create rollup jobs

        2. Delete rollup jobs

        3. Get job

        4. Get rollup caps

        5. Get rollup index caps

        6. Rollup search

        7. Rollup job configuration

        8. Start rollup jobs

        9. Stop rollup jobs

      17. Search APIs

        1. Search

        2. URI Search

        3. Request Body Search

        4. Search Template

        5. Multi Search Template

        6. Search Shards API

        7. Suggesters

        8. Multi Search API

        9. Count API

        10. Validate API

        11. Explain API

        12. Profile API

        13. Field Capabilities API

        14. Ranking Evaluation API

      18. Security APIs

        1. Authenticate

        2. Change passwords

        3. Clear cache

        4. Clear roles cache

        5. Create API keys

        6. Create or update application privileges

        7. Create or update role mappings

        8. Create or update roles

        9. Create or update users

        10. Delegate PKI authentication

        11. Delete application privileges

        12. Delete role mappings

        13. Delete roles

        14. Delete users

        15. Disable users

        16. Enable users

        17. Get API key information

        18. Get application privileges

        19. Get builtin privileges

        20. Get role mappings

        21. Get roles

        22. Get token

        23. Get users

        24. Has privileges

        25. Invalidate API key

        26. Invalidate token

        27. OpenID Connect Prepare Authentication API

        28. OpenID Connect authenticate API

        29. OpenID Connect logout API

        30. SSL certificate

      19. Snapshot lifecycle management API

        1. Put snapshot lifecycle policy

        2. Get snapshot lifecycle policy

        3. Execute snapshot lifecycle policy

        4. Delete snapshot lifecycle policy

      20. Transform APIs

        1. Create transforms

        2. Update transforms

        3. Delete transforms

        4. Get transforms

        5. Get transform statistics

        6. Preview transforms

        7. Start transforms

        8. Stop transforms

      21. Watcher APIs

        1. Ack watch

        2. Activate watch

        3. Deactivate watch

        4. Delete watch

        5. Execute watch

        6. Get watch

        7. Get Watcher stats

        8. Put watch

        9. Start watch service

        10. Stop watch service

      22. Definitions

        1. Datafeed resources

        2. Data frame analytics job resources

        3. Data frame analytics evaluation resources

        4. Job resources

        5. Job statistics

        6. Model snapshot resources

        7. Role mapping resources

        8. Results resources

        9. Transform resources

  2. Logstash

    1. Logstash Introduction

    2. Getting Started with Logstash

      1. Installing Logstash

      2. Stashing Your First Event

      3. Parsing Logs with Logstash

      4. Stitching Together Multiple Input and Output Plugins

    3. How Logstash Works

      1. Execution Model

    4. Setting Up and Running Logstash

      1. Logstash Directory Layout

      2. Logstash Configuration Files

      3. logstash.yml

      4. Secrets keystore for secure settings

      5. Running Logstash from the Command Line

      6. Running Logstash as a Service on Debian or RPM

      7. Running Logstash on Docker

      8. Configuring Logstash for Docker

      9. Running Logstash on Windows

      10. Logging

      11. Shutting Down Logstash

      12. Setting Up X-Pack

    5. Upgrading Logstash

      1. Upgrading Using Package Managers

      2. Upgrading Using a Direct Download

      3. Upgrading between minor versions

      4. Upgrading Logstash to 7.0

      5. Upgrading with the Persistent Queue Enabled

    6. Configuring Logstash

      1. Structure of a Config File

      2. Accessing Event Data and Fields in the Configuration

      3. Using Environment Variables in the Configuration

      4. Logstash Configuration Examples

      5. Multiple Pipelines

      6. Pipeline-to-Pipeline Communication

      7. Reloading the Config File

      8. Managing Multiline Events

      9. Glob Pattern Support

      10. Converting Ingest Node Pipelines

      11. Logstash-to-Logstash Communication

      12. Centralized Pipeline Management

      13. X-Pack security

      14. X-Pack Settings

    7. Managing Logstash

      1. Centralized Pipeline Management

    8. Working with Logstash Modules

      1. Using Elastic Cloud

      2. ArcSight Module

      3. Netflow Module (deprecated)

      4. Azure Module

    9. Working with Filebeat Modules

      1. Use ingest pipelines for parsing

      2. Use Logstash pipelines for parsing

      3. Example: Set up Filebeat modules to work with Kafka and Logstash

    10. Data Resiliency

      1. Persistent Queues

      2. Dead Letter Queues

    11. Transforming Data

      1. Performing Core Operations

      2. Deserializing Data

      3. Extracting Fields and Wrangling Data

      4. Enriching Data with Lookups

    12. Deploying and Scaling Logstash

    13. Performance Tuning

      1. Performance Troubleshooting Guide

      2. Tuning and Profiling Logstash Performance

    14. Monitoring Logstash with APIs

      1. Node Info API

      2. Plugins Info API

      3. Node Stats API

      4. Hot Threads API

    15. Monitoring Logstash with X-Pack

      1. Metricbeat collection

      2. Internal collection

      3. Monitoring UI

      4. Pipeline Viewer UI

      5. Troubleshooting

    16. Working with plugins

      1. Generating Plugins

      2. Offline Plugin Management

      3. Private Gem Repositories

      4. Event API

    17. Input plugins

      1. azure_event_hubs

      2. beats

      3. cloudwatch

      4. couchdb_changes

      5. dead_letter_queue

      6. elasticsearch

      7. exec

      8. file

      9. ganglia

      10. gelf

      11. generator

      12. github

      13. google_cloud_storage

      14. google_pubsub

      15. graphite

      16. heartbeat

      17. http

      18. http_poller

      19. imap

      20. irc

      21. java_generator

      22. java_stdin

      23. jdbc

      24. jms

      25. jmx

      26. kafka

      27. kinesis

      28. log4j

      29. lumberjack

      30. meetup

      31. pipe

      32. puppet_facter

      33. rabbitmq

      34. redis

      35. relp

      36. rss

      37. s3

      38. salesforce

      39. snmp

      40. snmptrap

      41. sqlite

      42. sqs

      43. stdin

      44. stomp

      45. syslog

      46. tcp

      47. twitter

      48. udp

      49. unix

      50. varnishlog

      51. websocket

      52. wmi

      53. xmpp

    18. Output plugins

      1. boundary

      2. circonus

      3. cloudwatch

      4. csv

      5. datadog

      6. datadog_metrics

      7. elastic_app_search

      8. elasticsearch

      9. email

      10. exec

      11. file

      12. ganglia

      13. gelf

      14. google_bigquery

      15. google_cloud_storage

      16. google_pubsub

      17. graphite

      18. graphtastic

      19. http

      20. influxdb

      21. irc

      22. java_sink

      23. java_stdout

      24. juggernaut

      25. kafka

      26. librato

      27. loggly

      28. lumberjack

      29. metriccatcher

      30. mongodb

      31. nagios

      32. nagios_nsca

      33. opentsdb

      34. pagerduty

      35. pipe

      36. rabbitmq

      37. redis

      38. redmine

      39. riak

      40. riemann

      41. s3

      42. sns

      43. solr_http

      44. sqs

      45. statsd

      46. stdout

      47. stomp

      48. syslog

      49. tcp

      50. timber

      51. udp

      52. webhdfs

      53. websocket

      54. xmpp

      55. zabbix

    19. Filter plugins

      1. aggregate

      2. alter

      3. bytes

      4. cidr

      5. cipher

      6. clone

      7. csv

      8. date

      9. de_dot

      10. dissect

      11. dns

      12. drop

      13. elapsed

      14. elasticsearch

      15. environment

      16. extractnumbers

      17. fingerprint

      18. geoip

      19. grok

      20. http

      21. i18n

      22. java_uuid

      23. jdbc_static

      24. jdbc_streaming

      25. json

      26. json_encode

      27. kv

      28. memcached

      29. metricize

      30. metrics

      31. mutate

      32. prune

      33. range

      34. ruby

      35. sleep

      36. split

      37. syslog_pri

      38. threats_classifier

      39. throttle

      40. tld

      41. translate

      42. truncate

      43. urldecode

      44. useragent

      45. uuid

      46. xml

    20. Codec plugins

      1. avro

      2. cef

      3. cloudfront

      4. cloudtrail

      5. collectd

      6. dots

      7. edn

      8. edn_lines

      9. es_bulk

      10. fluent

      11. graphite

      12. gzip_lines

      13. jdots

      14. java_line

      15. java_plain

      16. json

      17. json_lines

      18. line

      19. msgpack

      20. multiline

      21. netflow

      22. nmap

      23. plain

      24. protobuf

      25. rubydebug

    21. Tips and Best Practices

    22. Troubleshooting Common Problems

    23. Contributing to Logstash

      1. How to write a Logstash input plugin

      2. How to write a Logstash codec plugin

      3. How to write a Logstash filter plugin

      4. How to write a Logstash output plugin

      5. Documenting your plugin

      6. Contributing a Patch to a Logstash Plugin

      7. Logstash Plugins Community Maintainer Guide

      8. Submitting your plugin to RubyGems.org and the logstash-plugins repository

    24. Contributing a Java Plugin

      1. How to write a Java input plugin

      2. How to write a Java codec plugin

      3. How to write a Java filter plugin

      4. How to write a Java output plugin

    25. Glossary of Terms

  3. Kibana

    1. Introduction

    2. Set Up Kibana

      1. Installing Kibana

      2. Install Kibana with .tar.gz

      3. Install Kibana with Debian Package

      4. Install Kibana with RPM

      5. Install Kibana on Windows

      6. Install Kibana on macOS with Homebrew

    3. Starting and stopping Kibana

    4. Configuring Kibana

      1. APM settings

      2. Code settings

      3. Development tools settings

      4. Graph settings

      5. Infrastructure UI settings

      6. i18n settings in Kibana

      7. Logs UI settings

      8. Machine learning settings

      9. Monitoring settings

      10. Reporting settings

      11. Secure settings

      12. Security settings

      13. Spaces settings

    5. Running Kibana on Docker

    6. Accessing Kibana

    7. Connect Kibana with Elasticsearch

    8. Using Kibana in a production environment

    9. Upgrading Kibana

      1. Standard upgrade

      2. Troubleshooting saved object migrations

    10. Configuring monitoring

      1. Collecting monitoring data

      2. Collecting monitoring data with Metricbeat

      3. Viewing monitoring data

    11. Configuring security

      1. Authentication

      2. Encrypting communications

      3. Audit Logging

    12. Getting Started

      1. Add sample data

      2. Explore Kibana using sample data

      3. Build your own dashboard

        1. Define your index patterns

        2. Discover your data

        3. Visualize your data

        4. Add visualizations to a dashboard

    13. Discover

      1. Setting the time filter

      2. Searching your data

        1. Kibana Query Language

        2. Lucene query syntax

        3. Saving searches

        4. Saving queries

        5. Change the indices you’re searching

        6. Refresh the search results

      3. Filtering by Field

      4. Viewing Document Data

      5. Viewing Document Context

      6. Viewing Field Data Statistics

    14. Visualize

      1. Creating a Visualization

      2. Saving Visualizations

      3. Using rolled up data in a visualization

      4. Line, Area, and Bar charts

      5. Controls Visualization

        1. Adding Input Controls

        2. Global Options

      6. Data Table

      7. Markdown Widget

      8. Metric

      9. Goal and Gauge

      10. Pie Charts

      11. Coordinate Maps

      12. Region Maps

      13. Timelion

      14. TSVB

      15. Tag Clouds

      16. Heatmap Chart

      17. Vega Graphs

        1. Getting Started with Vega

        2. Vega vs Vega-Lite

        3. Querying Elasticsearch

        4. Elastic Map Files

        5. Vega with a Map

        6. Debugging

        7. Useful Links

      18. Inspecting Visualizations

    15. Dashboard

      1. Create a dashboard

      2. Dashboard-only mode

    16. Canvas

      1. Canvas tutorial

      2. Create a workpad

      3. Showcase your data with elements

      4. Present your workpad

      5. Share your workpad

      6. Canvas function reference

        1. TinyMath functions

    17. Extend your use case

      1. Graph data connections

        1. Using Graph

        2. Configuring Graph

        3. Troubleshooting

        4. Limitations

      2. Machine learning

    18. Elastic Maps

      1. Getting started with Elastic Maps

        1. Creating a new map

        2. Adding a choropleth layer

        3. Adding layers for Elasticsearch data

        4. Saving the map

        5. Adding the map to a dashboard

      2. Heat map layer

      3. Tile layer

      4. Vector layer

        1. Vector styling

        2. Vector style properties

        3. Vector tooltips

      5. Plot big data without plotting too much data

        1. Grid aggregation

        2. Most recent entities

        3. Point to point

        4. Term join

      6. Searching your data

        1. Creating filters from your map

        2. Filtering a single layer

        3. Searching across multiple indices

      7. Connecting to Elastic Maps Service

      8. Upload GeoJSON data

      9. Indexing GeoJSON data tutorial

      10. Elastic Maps troubleshooting

    19. Code

      1. Import your first repo

      2. Repo management

      3. Install language server

      4. Basic navigation

      5. Semantic code navigation

      6. Search

      7. Config for multiple Kibana instances

    20. Infrastructure

      1. Getting started with infrastructure monitoring

      2. Using the Infrastructure app

      3. Viewing infrastructure metrics

      4. Metrics Explorer

    21. Logs

      1. Getting started with logs monitoring

      2. Using the Logs app

      3. Configuring the Logs data

    22. APM

      1. Getting Started

      2. Visualizing Application Bottlenecks

      3. Using APM

        1. Filters

        2. Services overview

        3. Traces overview

        4. Transaction overview

        5. Span timeline

        6. Errors overview

        7. Metrics overview

        8. Machine Learning integration

        9. APM Agent configuration

        10. Advanced queries

    23. Uptime

      1. Overview

      2. Monitor

    24. SIEM

      1. Using the SIEM UI

      2. Anomaly Detection with Machine Learning

    25. Dev Tools

      1. Console

      2. Profiling queries and aggregations

        1. Getting Started

        2. Profiling a more complicated query

        3. Rendering pre-captured profiler JSON

      3. Debugging grok expressions

    26. Stack Monitoring

      1. Beats Metrics

      2. Cluster Alerts

      3. Elasticsearch Metrics

      4. Kibana Metrics

      5. Logstash Metrics

      6. Troubleshooting

    27. Management

      1. License Management

      2. Index patterns

        1. Cross-cluster search

      3. Rollup jobs

      4. Index lifecycle policies

        1. Creating an index lifecycle policy

        2. Managing index lifecycle policies

        3. Adding a policy to an index

        4. Example of using an index lifecycle policy

      5. Managing Fields

        1. String Field Formatters

        2. Date Field Formatters

        3. Geographic Point Field Formatters

        4. Numeric Field Formatters

        5. Scripted Fields

      6. Index management

      7. Setting advanced options

      8. Saved objects

      9. Managing Beats

      10. Working with remote clusters

      11. Snapshot and Restore

      12. Spaces

      13. Security

        1. Granting access to Kibana

        2. Kibana role management

        3. Kibana privileges

      14. Watcher

      15. Upgrade Assistant

    28. Reporting from Kibana

      1. Automating report generation

      2. PDF layout modes

      3. Reporting configuration

        1. Reporting and security

        2. Secure the reporting endpoints

        3. Chromium sandbox

      4. Troubleshooting

      5. Reporting integration

    29. REST API

      1. Features API

        1. Get features

      2. Kibana Spaces APIs

        1. Create space

        2. Update space

        3. Get space

        4. Get all spaces

        5. Delete space

        6. Copy saved objects to space

        7. Resolve copy to space conflicts

      3. Kibana role management APIs

        1. Create or update role

        2. Get specific role

        3. Get all roles

        4. Delete role

      4. Saved objects APIs

        1. Get object

        2. Bulk get objects

        3. Find objects

        4. Create object

        5. Bulk create objects

        6. Update object

        7. Delete object

        8. Export objects

        9. Import objects

        10. Resolve import errors

      5. Dashboard import and export APIs

        1. Import dashboard

        2. Dashboard export

      6. Logstash configuration management APIs

        1. Create pipeline

        2. Retrieve pipeline

        3. Delete pipeline

        4. List pipeline

      7. URL shortening API

        1. Shorten URL

      8. Upgrade assistant APIs

        1. Upgrade readiness status

        2. Start or resume reindex

        3. Check reindex status

        4. Cancel reindex

    30. Kibana plugins

      1. Install plugins

      2. Update and remove plugins

      3. Disable plugins

      4. Configure the plugin manager

      5. Known Plugins

    31. Limitations

      1. Nested Objects

      2. Exporting data

    32. Developer guide

      1. Core Development

        1. Considerations for basePath

        2. Managing Dependencies

        3. Modules and Autoloading

        4. Communicating with Elasticsearch

        5. Unit Testing

        6. Functional Testing

      2. Plugin Development

        1. Plugin Resources

        2. UI Exports

        3. Plugin feature registration

        4. Functional Tests for Plugins

        5. Localization for plugins

      3. Developing Visualizations

        1. Embedding Visualizations

        2. Developing Visualizations

        3. Visualization Factory

        4. Visualization Editors

        5. Visualization Request Handlers

        6. Visualization Response Handlers

        7. Vis object

        8. AggConfig object

      4. Add Data Guide

      5. Security

        1. Role-based access control

      6. Pull request review guidelines

      7. Interpreting CI Failures

  4. Beats Platform

    1. Community Beats

    2. Getting started with Beats

    3. Config file format

      1. Namespacing

      2. Config file data types

      3. Environment variables

      4. Reference variables

      5. Config file ownership and permissions

      6. Command line arguments

      7. YAML tips and gotchas

    4. Upgrading

      1. Upgrade between minor versions

      2. Upgrade from 6.x to 7.x

      3. Troubleshooting Beats upgrade issues

  5. Beats Developer Guide

    1. Contributing to Beats

    2. Community Beats

    3. Creating a New Beat

      1. Getting Ready

      2. Overview

      3. Generating Your Beat

      4. Fetching Dependencies and Setting up the Beat

      5. Building and Running the Beat

      6. The Beater Interface

      7. Sharing Your Beat with the Community

      8. Naming Conventions

    4. Creating New Kibana Dashboards

      1. Importing Existing Beat Dashboards

      2. Building Your Own Beat Dashboards

      3. Generating the Beat Index Pattern

      4. Exporting New and Modified Beat Dashboards

      5. Archiving Your Beat Dashboards

      6. Sharing Your Beat Dashboards

    5. Adding a New Protocol to Packetbeat

      1. Getting Ready

      2. Protocol Modules

      3. Testing

    6. Extending Metricbeat

      1. Overview

      2. Creating a Metricset

      3. Metricset Details

      4. Creating a Metricbeat Module

      5. Creating a Beat based on Metricbeat

      6. Metricbeat Developer FAQ

    7. Creating a New Filebeat Module

    8. Migrating dashboards from Kibana 5.x to 6.x

  6. Filebeat

    1. Overview

    2. Getting Started With Filebeat

      1. Step 1: Install Filebeat

      2. Step 2: Configure Filebeat

      3. Step 3: Load the index template in Elasticsearch

      4. Step 4: Set up the Kibana dashboards

      5. Step 5: Start Filebeat

      6. Step 6: View the sample Kibana dashboards

      7. Quick start: modules for common log formats

      8. Repositories for APT and YUM

    3. Setting up and running Filebeat

      1. Directory layout

      2. Secrets keystore

      3. Command reference

      4. Running Filebeat on Docker

      5. Running Filebeat on Kubernetes

      6. Filebeat and systemd

      7. Stopping Filebeat

    4. Upgrading Filebeat

    5. How Filebeat works

    6. Configuring Filebeat

      1. Specify which modules to run

      2. Configure inputs

      3. Manage multiline messages

      4. Specify general settings

      5. Load external configuration files

      6. Configure the internal queue

      7. Configure the output

      8. Configure index lifecycle management

      9. Load balance the output hosts

      10. Specify SSL settings

      11. Filter and enhance the exported data

      12. Parse data by using ingest node

      13. Enrich events with geoIP information

      14. Configure project paths

      15. Configure the Kibana endpoint

      16. Load the Kibana dashboards

      17. Load the Elasticsearch index template

      18. Configure logging

      19. Use environment variables in the configuration

      20. Autodiscover

      21. YAML tips and gotchas

      22. Regular expression support

      23. HTTP Endpoint

      24. filebeat.reference.yml

    7. Beats central management

      1. How central management works

      2. Enroll Beats in central management

    8. Modules

      1. Modules overview

      2. Apache module

      3. Auditd module

      4. AWS module

      5. CEF module

      6. Cisco module

      7. Coredns Module

      8. Elasticsearch module

      9. Envoyproxy Module

      10. Google Cloud module

      11. haproxy module

      12. IBM MQ module

      13. Icinga module

      14. IIS module

      15. Iptables module

      16. Kafka module

      17. Kibana module

      18. Logstash module

      19. MongoDB module

      20. MSSQL module

      21. MySQL module

      22. nats module

      23. NetFlow module

      24. Nginx module

      25. Osquery module

      26. Palo Alto Networks module

      27. PostgreSQL module

      28. RabbitMQ module

      29. Redis module

      30. Santa module

      31. Suricata module

      32. System module

      33. Traefik module

      34. Zeek (Bro) Module

    9. Exported fields

      1. Apache fields

      2. Auditd fields

      3. AWS fields

      4. Beat fields

      5. Decode CEF processor fields fields

      6. CEF fields

      7. Cisco fields

      8. Cloud provider metadata fields

      9. Coredns fields

      10. Docker fields

      11. ECS fields

      12. elasticsearch fields

      13. Envoyproxy fields

      14. Google Cloud fields

      15. haproxy fields

      16. Host fields

      17. ibmmq fields

      18. Icinga fields

      19. IIS fields

      20. iptables fields

      21. Jolokia Discovery autodiscover provider fields

      22. Kafka fields

      23. kibana fields

      24. Kubernetes fields

      25. Log file content fields

      26. logstash fields

      27. mongodb fields

      28. mssql fields

      29. MySQL fields

      30. nats fields

      31. NetFlow fields

      32. NetFlow fields

      33. Nginx fields

      34. Osquery fields

      35. panw fields

      36. PostgreSQL fields

      37. Process fields

      38. RabbitMQ fields

      39. Redis fields

      40. s3 fields

      41. Google Santa fields

      42. Suricata fields

      43. System fields

      44. Traefik fields

      45. Zeek fields

    10. Monitoring Filebeat

      1. Internal collection

        1. Settings for internal monitoring collection

      2. Metricbeat collection

    11. Securing Filebeat

      1. Secure communication with Elasticsearch

      2. Secure communication with Logstash

      3. Use X-Pack security

        1. Grant users access to secured resources

        2. Configure authentication credentials

        3. Configure Filebeat to use encrypted connections

      4. Use Linux Secure Computing Mode (seccomp)

    12. Troubleshooting

      1. Get help

      2. Debug

      3. Common problems

        1. Can’t read log files from network volumes

        2. Filebeat isn’t collecting lines from a file

        3. Too many open file handlers

        4. Registry file is too large

        5. Inode reuse causes Filebeat to skip lines

        6. Log rotation results in lost or duplicate events

        7. Open file handlers cause issues with Windows file rotation

        8. Filebeat is using too much CPU

        9. Dashboard in Kibana is breaking up data fields incorrectly

        10. Fields are not indexed or usable in Kibana visualizations

        11. Filebeat isn’t shipping the last line of a file

        12. Filebeat keeps open file handlers of deleted files for a long time

        13. Filebeat uses too much bandwidth

        14. Error loading config file

        15. Found unexpected or unknown characters

        16. Logstash connection doesn’t work

        17. @metadata is missing in Logstash

        18. Not sure whether to use Logstash or Beats

        19. SSL client fails to connect to Logstash

        20. Monitoring UI shows fewer Beats than expected

    13. A. Contributing to Beats

 

:

[Logstash] JSON filter plugin

Elastic/Logstash 2019. 11. 4. 14:16

공홈에 올라와 있는 문서의 번역 본 정도로 정리를 해보려고 합니다.

별거 아니지만 JSON filter 를 많이 사용하면서 Validation 에 대한 인식이 부족해서 오류를 발생 시키는 경우가 꽤 많이 있습니다.

기억력을 돕기 위해 작성해 봅니다.

 

공식문서)

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

 

이 내용은 logstash reference 문서 내 filter 항목에 해당 합니다.

용도는 말 그대로 입니다. JSON  parsing 을 하는 filter 입니다.

 

여기서 문제가 들어 오는 JSON 데이터가 항상 validate 할 거라고 생각 하고 구현 하시는 분들이 계시는데 이 부분이 문제가 됩니다.

기본 이라고 생각 하지만 validation 에 대한 개발자의 생각과 경험이 요즘은 다른 것 같더라구요.

 

암튼, 그래서 JSON filter 사용 시 제공하는 Option 에 대해서 숙지 하고 사용하시면 좋겠습니다.

기본적으로는 Common Options 를 먼저 보시는게 좋습니다.

 

JSON Filter Configuration Options)

Setting Input type Required
skip_on_invalid_json boolean No
source string Yes
tag_on_failure array No
target string No
  • skip_on_invalid_json
    json 이 아닌 데이터가 들어 올 경우 에러를 발생 시키지 않고 skip 시키기 위한 옵션 입니다.
    기본 설정 값이 false 이기 때문에 잘 못된 데이터에 대해서 오류가 발생 하게 됩니다.

  • source
    json parsing 을 하기 위한 field 를 지정 하게 됩니다.

  • tag_on_failure
    정상적으로 처리 되지 않았을 경우 tags 라는 filed 에 "_jsonparsefailure" 값이 추가 됩니다.

  •  target
    source field 내 json value 가 target filed 로 등록 되며, 이미 target field 가 있다면 overwrite 됩니다.

위 설정 중에서 skip_on_invalid_json  과 tag_on_failure 만 잘 설정 하셔도 invalid data 에 대한 오류는 잘 넘길 수 있습니다.

간혹 이 오류로 인해서 logstash 가 먹통이 되는 걸 예방 할 수 있기 때문 입니다.

:

[Elasticsearch] 앱 내 사용자 행동로그 수집 파이프라인 구성

Elastic/Elasticsearch 2019. 10. 17. 15:13

사용하고자 하는 Software Stack 은 다양하게 많이 있습니다.

일반적으로 아래 파이프라인으로 많이들 구성 합니다.

 

1. App -> Stream service -> Consumer -> Elasticsearch
2. App -> Stream service -> Producer -> Queue -> Consumer -> Elasticsearch
3. App -> Logging service (daemon, http, file ...) -> Consumer -> Elasticsearch
4. App -> Logging service (daemon, http, file ...) -> Producer -> Queue -> Consumer -> Elasticsearch

 

이걸 다시 Elastic Stack 으로 변환 하면

 

Producer 는)

- Filebeat

- Logstash

 

Queue 는)

- Logstash persistent queue

 

Consumer 는)

- Logstash

 

이 외에도 sqs, dynamodb, redis, kafka, fluentd, storm 등 활용 가능한 오픈소스들이 많이 준비되어 있습니다.

 

가장 쉽고 일반적인 구성이라고 보시면 될 것 같습니다.

:

[Logstash] --config.reload.automatic 사용 경험 공유

Elastic/Logstash 2018. 10. 4. 11:28

Logstash 사용 시 --config.reload.automatic 설정을 통해서 conf 파일에 대한 변경 사항을 데몬 재시작 없이 할 수 있습니다.

하지만 모든 변경 사항에 대해서 반영이 가능 하지 않기 때문에 사용 시 주의 하셔야 합니다.


크게 사용 방법은 두 가지로 나뉩니다.

1. --config.reload.automatic 을 통한 자동 갱신

2. Logstash 재시작을 통한 갱신


2번의 과정을 하고 싶지 않기 때문에 1번을 설정해서 사용을 하는데 문제는 이게 모든 plugins 에서 동작 하지는 않는 다는 것입니다.


만약 아래와 같은 에러를 접하셨다면, 해당 plugin 또는 pipeline 은 auto reload 설정이 동작 하지 않는 것들 이니 참고 하시기 바랍니다.


[에러내용]

[ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>“Cannot reload pipeline, because the existing pipeline is not reloadable”, :backtrace=>nil}


쉽게는 기본적인 syntax 오류 같은건 바로 바로 반영 됩니다. :)

:

[Logstash] AWS SQS + Logstash 구성

Elastic/Logstash 2018. 8. 28. 09:52

가끔 사용하다가도 처음 사용할  때 이해했던 내용과 다르게 기억 될 때가 있습니다.

그래서 또 복습 합니다.


구성은 아래와 같습니다.


File Log --> Logstash Agent --> SQS -->

Logstash Collector --> 

File

Elasticsearch

Logstash Collector --> 

File

Elasticsearch


각 서버의 file log 를 input file 로 읽고 output sqs 로 보냅니다.

그런 후 input sqs 를 읽고 multi output 으로 file 과 elasticsearch 로 저장을 하게 되는 구조 입니다.


여기서 AWS 의 SQS 에 대한 메시지 수명 주기에 대한 이해가 필요 합니다.


원문)

https://aws.amazon.com/ko/sqs/details/


Amazon SQS 메시지 수명 주기

Amazon SQS에 저장된 메시지에는 관리가 쉬우면서도 모든 메시지가 처리되도록 보장하는 수명 주기가 있습니다.


1. 메시지를 보내야 하는 시스템에서는 Amazon SQS 대기열을 선택하고 SendMessage를 사용하여 새 메시지를 전송합니다.

2. 메시지를 처리하는 다른 시스템은 처리할 메시지가 더 많이 필요해지므로 ReceiveMessage를 호출하고, 해당 메시지가 반환됩니다.

3. ReceiveMessage에 의해 메시지가 반환되면 해당 메시지는 제한 시간이 초과할 때까지는 다른 어떤 ReceiveMessage에 의해서도 반환되지 않습니다. 이는 다수의 소비자가 동일한 메시지를 동시에 처리하는 것을 방지합니다.

4. 메시지를 처리하는 시스템에서 메시지 작업을 성공적으로 완료하면, 해당 시스템에서는 다른 시스템에서 해당 메시지를 다시 처리하지 않도록 DeleteMessage를 호출하여 메시지를 대기열에서 제거합니다. 시스템에서 메시지 처리에 실패하는 경우, 제한 시간이 초과하는 즉시 다른 ReceiveMessage 호출을 통해 해당 메시지를 읽습니다.

5. 소스 대기열에 배달 못한 편지 대기열이 연결되어 있는 경우 지정한 최대 배달 시도 횟수에 도달한 이후에는 메시지가 배달 못한 편지 대기열로 이동됩니다.


결국 정리 하면, consumer 가 읽어 가면 다른 consumer 는 이미 읽어간 데이터를 읽어 가지 못합니다. 이유는 위 설명에서와 같이 삭제 되기 때문 입니다.

단일 큐를 사용하면서 큐에 쌓인 메시지의 소비는 다중 consumer 로 처리 하고 단일 저장소로 저장 할 경우 쉽게 구성 할 수 있다는 이야기 였습니다.


여기서 제가 착각한 포인트는 읽어간 메시지가 삭제 되지 않는다는 것이였구요.

이런 오해는 "메시지는 최대 14일 동안 대기열에 보관됩니다." 라는 걸 보고 착각한 것이였습니다.


:

[Beats] 오랜만에 Filebeat 설치

Elastic/Beats 2018. 7. 25. 20:23

설치환경)

AWS EC2

Ubuntu


다운로드)

https://www.elastic.co/kr/downloads/beats/filebeat

https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.1-linux-x86_64.tar.gz


다운 받으시고 그냥 압축 해제 하시면 됩니다.


tree 는 그냥 제 맥북 기준으로 보여 드립니다.


filebeat-6.3.2-darwin-x86_64

├── LICENSE.txt

├── NOTICE.txt

├── README.md

├── fields.yml

├── filebeat

├── filebeat.reference.yml

├── filebeat.yml

├── kibana

│   ├── 5

│   │   ├── dashboard

│   │   │   ├── 0d3f2380-fa78-11e6-ae9b-81e5311e8cab.json

│   │   │   ├── 26309570-2419-11e7-a83b-d5f4cebac9ff.json

│   │   │   ├── 277876d0-fa2c-11e6-bbd3-29c986c96e5a.json

│   │   │   ├── 5517a150-f9ce-11e6-8115-a7c18106d86a.json

│   │   │   ├── 7fea2930-478e-11e7-b1f0-cb29bac6bf8b.json

│   │   │   ├── Filebeat-Apache2-Dashboard.json

│   │   │   ├── Filebeat-MySQL-Dashboard.json

│   │   │   ├── Filebeat-Nginx-Dashboard.json

│   │   │   ├── Filebeat-Traefik-Dashboard.json

│   │   │   ├── Filebeat-syslog-dashboard.json

│   │   │   ├── ML-Nginx-Access-Remote-IP-Count-Explorer.json

│   │   │   ├── ML-Nginx-Remote-IP-URL-Explorer.json

│   │   │   ├── ML-Traefik-Access-Remote-IP-Count-Explorer.json

│   │   │   ├── ML-Traefik-Remote-IP-URL-Explorer.json

│   │   │   ├── b9163ea0-2417-11e7-a83b-d5f4cebac9ff.json

│   │   │   ├── dfbb49f0-0a0f-11e7-8a62-2d05eaaac5cb.json

│   │   │   └── f693d260-2417-11e7-a83b-d5f4cebac9ff.json

│   │   ├── index-pattern

│   │   │   └── filebeat.json

│   │   ├── search

│   │   │   ├── 0ab87b80-478e-11e7-b1f0-cb29bac6bf8b.json

│   │   │   ├── 4ac0a370-0a11-11e7-8b04-eb22a5669f27.json

│   │   │   ├── 62439dc0-f9c9-11e6-a747-6121780e0414.json

│   │   │   ├── 710043e0-2417-11e7-a83b-d5f4cebac9ff.json

│   │   │   ├── 73613570-4791-11e7-be88-2ddb32f3df97.json

│   │   │   ├── 8030c1b0-fa77-11e6-ae9b-81e5311e8cab.json

│   │   │   ├── Apache2-access-logs.json

│   │   │   ├── Apache2-errors-log.json

│   │   │   ├── Filebeat-MySQL-Slow-log.json

│   │   │   ├── Filebeat-MySQL-error-log.json

│   │   │   ├── Filebeat-Nginx-module.json

│   │   │   ├── Filebeat-Traefik-module.json

│   │   │   ├── ML-Filebeat-Nginx-Access.json

│   │   │   ├── ML-Filebeat-Traefik-Access.json

│   │   │   ├── Syslog-system-logs.json

│   │   │   ├── b6f321e0-fa25-11e6-bbd3-29c986c96e5a.json

│   │   │   ├── c876e6a0-2418-11e7-a83b-d5f4cebac9ff.json

│   │   │   ├── eb0039f0-fa7f-11e6-a1df-a78bd7504d38.json

│   │   │   └── ffaf5a30-2413-11e7-a0d9-39604d45ca7f.json

│   │   └── visualization

│   │       ├── 0bc34b60-2419-11e7-a83b-d5f4cebac9ff.json

│   │       ├── 12667040-fa80-11e6-a1df-a78bd7504d38.json

│   │       ├── 2bb0fa70-0a11-11e7-9e84-43da493ad0c7.json

│   │       ├── 2cf77780-2418-11e7-a83b-d5f4cebac9ff.json

│   │       ├── 341ffe70-f9ce-11e6-8115-a7c18106d86a.json

│   │       ├── 346bb290-fa80-11e6-a1df-a78bd7504d38.json

│   │       ├── 3cec3eb0-f9d3-11e6-8a3e-2b904044ea1d.json

│   │       ├── 51164310-fa2b-11e6-bbd3-29c986c96e5a.json

│   │       ├── 5c7af030-fa2a-11e6-bbd3-29c986c96e5a.json

│   │       ├── 5dd15c00-fa78-11e6-ae9b-81e5311e8cab.json

│   │       ├── 5ebdbe50-0a0f-11e7-825f-6748cda7d858.json

│   │       ├── 6295bdd0-0a0e-11e7-825f-6748cda7d858.json

│   │       ├── 78b74f30-f9cd-11e6-8115-a7c18106d86a.json

│   │       ├── 78b9afe0-478f-11e7-b1f0-cb29bac6bf8b.json

│   │       ├── Apache2-access-unique-IPs-map.json

│   │       ├── Apache2-browsers.json

│   │       ├── Apache2-error-logs-over-time.json

│   │       ├── Apache2-operating-systems.json

│   │       ├── Apache2-response-codes-of-top-URLs.json

│   │       ├── Apache2-response-codes-over-time.json

│   │       ├── Errors-over-time.json

│   │       ├── ML-Nginx-Access-Map.json

│   │       ├── ML-Nginx-Access-Remote-IP-Timechart.json

│   │       ├── ML-Nginx-Access-Response-Code-Timechart.json

│   │       ├── ML-Nginx-Access-Top-Remote-IPs-Table.json

│   │       ├── ML-Nginx-Access-Top-URLs-Table.json

│   │       ├── ML-Nginx-Access-Unique-Count-URL-Timechart.json

│   │       ├── ML-Traefik-Access-Map.json

│   │       ├── ML-Traefik-Access-Remote-IP-Timechart.json

│   │       ├── ML-Traefik-Access-Response-Code-Timechart.json

│   │       ├── ML-Traefik-Access-Top-Remote-IPs-Table.json

│   │       ├── ML-Traefik-Access-Top-URLs-Table.json

│   │       ├── ML-Traefik-Access-Unique-Count-URL-Timechart.json

│   │       ├── MySQL-Error-logs-levels.json

│   │       ├── MySQL-Slow-logs-by-count.json

│   │       ├── MySQL-Slow-queries-over-time.json

│   │       ├── MySQL-error-logs.json

│   │       ├── MySQL-slowest-queries.json

│   │       ├── New-Visualization.json

│   │       ├── Nginx-Access-Browsers.json

│   │       ├── Nginx-Access-Map.json

│   │       ├── Nginx-Access-OSes.json

│   │       ├── Nginx-Access-Response-codes-by-top-URLs.json

│   │       ├── Sent-sizes.json

│   │       ├── Syslog-events-by-hostname.json

│   │       ├── Syslog-hostnames-and-processes.json

│   │       ├── Traefik-Access-Browsers.json

│   │       ├── Traefik-Access-Map.json

│   │       ├── Traefik-Access-OSes.json

│   │       ├── Traefik-Access-Response-codes-by-top-URLs.json

│   │       ├── a59b5e00-2417-11e7-a83b-d5f4cebac9ff.json

│   │       ├── c5411910-0a87-11e7-8b04-eb22a5669f27.json

│   │       ├── d16bb400-f9cc-11e6-8115-a7c18106d86a.json

│   │       ├── d1726930-0a7f-11e7-8b04-eb22a5669f27.json

│   │       ├── d2864600-478f-11e7-be88-2ddb32f3df97.json

│   │       ├── d56ee420-fa79-11e6-a1df-a78bd7504d38.json

│   │       ├── d8e5dc40-2417-11e7-a83b-d5f4cebac9ff.json

│   │       ├── dc589770-fa2b-11e6-bbd3-29c986c96e5a.json

│   │       ├── dcccaa80-4791-11e7-be88-2ddb32f3df97.json

│   │       ├── e121b140-fa78-11e6-a1df-a78bd7504d38.json

│   │       ├── f398d2f0-fa77-11e6-ae9b-81e5311e8cab.json

│   │       └── fb09d4b0-2418-11e7-a83b-d5f4cebac9ff.json

│   └── 6

│       ├── dashboard

│       │   ├── Filebeat-Kafka-overview.json

│       │   ├── Filebeat-Mongodb-overview.json

│       │   ├── Filebeat-Postgresql-overview.json

│       │   ├── Filebeat-Postgresql-slowlogs.json

│       │   ├── Filebeat-apache2.json

│       │   ├── Filebeat-auditd.json

│       │   ├── Filebeat-auth-sudo-commands.json

│       │   ├── Filebeat-icinga-debug-log.json

│       │   ├── Filebeat-icinga-main-log.json

│       │   ├── Filebeat-icinga-startup-errors.json

│       │   ├── Filebeat-iis.json

│       │   ├── Filebeat-logstash-log.json

│       │   ├── Filebeat-logstash-slowlog.json

│       │   ├── Filebeat-mysql.json

│       │   ├── Filebeat-new-users-and-groups.json

│       │   ├── Filebeat-nginx-logs.json

│       │   ├── Filebeat-nginx-overview.json

│       │   ├── Filebeat-redis.json

│       │   ├── Filebeat-ssh-login-attempts.json

│       │   ├── Filebeat-syslog.json

│       │   ├── Filebeat-traefik-overview.json

│       │   ├── ml-nginx-access-remote-ip-count-explorer.json

│       │   ├── ml-nginx-remote-ip-url-explorer.json

│       │   ├── ml-traefik-access-remote-ip-count-explorer.json

│       │   ├── ml-traefik-remote-ip-url-explorer.json

│       │   ├── osquery-compliance.json

│       │   └── osquery-rootkit.json

│       └── index-pattern

│           └── filebeat.json

├── module

│   ├── apache2

│   │   ├── access

│   │   │   ├── config

│   │   │   │   └── access.yml

│   │   │   ├── ingest

│   │   │   │   └── default.json

│   │   │   └── manifest.yml

│   │   ├── error

│   │   │   ├── config

│   │   │   │   └── error.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   └── module.yml

│   ├── auditd

│   │   ├── log

│   │   │   ├── config

│   │   │   │   └── log.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   └── module.yml

│   ├── icinga

│   │   ├── debug

│   │   │   ├── config

│   │   │   │   └── debug.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   ├── main

│   │   │   ├── config

│   │   │   │   └── main.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   ├── module.yml

│   │   └── startup

│   │       ├── config

│   │       │   └── startup.yml

│   │       ├── ingest

│   │       │   └── pipeline.json

│   │       └── manifest.yml

│   ├── iis

│   │   ├── access

│   │   │   ├── config

│   │   │   │   └── iis-access.yml

│   │   │   ├── ingest

│   │   │   │   └── default.json

│   │   │   └── manifest.yml

│   │   └── error

│   │       ├── config

│   │       │   └── iis-error.yml

│   │       ├── ingest

│   │       │   └── default.json

│   │       └── manifest.yml

│   ├── kafka

│   │   ├── log

│   │   │   ├── config

│   │   │   │   └── log.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   └── module.yml

│   ├── logstash

│   │   ├── log

│   │   │   ├── config

│   │   │   │   └── log.yml

│   │   │   ├── ingest

│   │   │   │   ├── pipeline-json.json

│   │   │   │   └── pipeline-plain.json

│   │   │   └── manifest.yml

│   │   ├── module.yml

│   │   └── slowlog

│   │       ├── config

│   │       │   └── slowlog.yml

│   │       ├── ingest

│   │       │   ├── pipeline-json.json

│   │       │   └── pipeline-plain.json

│   │       └── manifest.yml

│   ├── mongodb

│   │   ├── log

│   │   │   ├── config

│   │   │   │   └── log.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   └── module.yml

│   ├── mysql

│   │   ├── error

│   │   │   ├── config

│   │   │   │   └── error.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   ├── module.yml

│   │   └── slowlog

│   │       ├── config

│   │       │   └── slowlog.yml

│   │       ├── ingest

│   │       │   └── pipeline.json

│   │       └── manifest.yml

│   ├── nginx

│   │   ├── access

│   │   │   ├── config

│   │   │   │   └── nginx-access.yml

│   │   │   ├── ingest

│   │   │   │   └── default.json

│   │   │   ├── machine_learning

│   │   │   │   ├── datafeed_low_request_rate.json

│   │   │   │   ├── datafeed_remote_ip_request_rate.json

│   │   │   │   ├── datafeed_remote_ip_url_count.json

│   │   │   │   ├── datafeed_response_code.json

│   │   │   │   ├── datafeed_visitor_rate.json

│   │   │   │   ├── low_request_rate.json

│   │   │   │   ├── remote_ip_request_rate.json

│   │   │   │   ├── remote_ip_url_count.json

│   │   │   │   ├── response_code.json

│   │   │   │   └── visitor_rate.json

│   │   │   └── manifest.yml

│   │   ├── error

│   │   │   ├── config

│   │   │   │   └── nginx-error.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   └── module.yml

│   ├── osquery

│   │   ├── module.yml

│   │   └── result

│   │       ├── config

│   │       │   └── result.yml

│   │       ├── ingest

│   │       │   └── pipeline.json

│   │       └── manifest.yml

│   ├── postgresql

│   │   ├── log

│   │   │   ├── config

│   │   │   │   └── log.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   └── module.yml

│   ├── redis

│   │   ├── log

│   │   │   ├── config

│   │   │   │   └── log.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   ├── module.yml

│   │   └── slowlog

│   │       ├── config

│   │       │   └── slowlog.yml

│   │       ├── ingest

│   │       │   └── pipeline.json

│   │       └── manifest.yml

│   ├── system

│   │   ├── auth

│   │   │   ├── config

│   │   │   │   └── auth.yml

│   │   │   ├── ingest

│   │   │   │   └── pipeline.json

│   │   │   └── manifest.yml

│   │   ├── module.yml

│   │   └── syslog

│   │       ├── config

│   │       │   └── syslog.yml

│   │       ├── ingest

│   │       │   └── pipeline.json

│   │       └── manifest.yml

│   └── traefik

│       ├── access

│       │   ├── config

│       │   │   └── traefik-access.yml

│       │   ├── ingest

│       │   │   └── pipeline.json

│       │   ├── machine_learning

│       │   │   ├── datafeed_low_request_rate.json

│       │   │   ├── datafeed_remote_ip_request_rate.json

│       │   │   ├── datafeed_remote_ip_url_count.json

│       │   │   ├── datafeed_response_code.json

│       │   │   ├── datafeed_visitor_rate.json

│       │   │   ├── low_request_rate.json

│       │   │   ├── remote_ip_request_rate.json

│       │   │   ├── remote_ip_url_count.json

│       │   │   ├── response_code.json

│       │   │   └── visitor_rate.json

│       │   ├── manifest.yml

│       │   └── tests

│       │       ├── test.log

│       │       └── test.log-expected.json

│       └── module.yml

└── modules.d

    ├── apache2.yml.disabled

    ├── auditd.yml.disabled

    ├── icinga.yml.disabled

    ├── iis.yml.disabled

    ├── kafka.yml.disabled

    ├── logstash.yml.disabled

    ├── mongodb.yml.disabled

    ├── mysql.yml.disabled

    ├── nginx.yml.disabled

    ├── osquery.yml.disabled

    ├── postgresql.yml.disabled

    ├── redis.yml.disabled

    ├── system.yml.disabled

    └── traefik.yml.disabled


97 directories, 254 files


도움말)

Usage:

  filebeat [flags]

  filebeat [command]


Available Commands:

  export      Export current config or index template

  help        Help about any command

  keystore    Manage secrets keystore

  modules     Manage configured modules

  run         Run filebeat

  setup       Setup index template, dashboards and ML jobs

  test        Test config

  version     Show current version info


Flags:

  -E, --E setting=value      Configuration overwrite

  -M, --M setting=value      Module configuration overwrite

  -N, --N                    Disable actual publishing for testing

  -c, --c string             Configuration file, relative to path.config (default "filebeat.yml")

      --cpuprofile string    Write cpu profile to file

  -d, --d string             Enable certain debug selectors

  -e, --e                    Log to stderr and disable syslog/file output

  -h, --help                 help for filebeat

      --httpprof string      Start pprof http server

      --memprofile string    Write memory profile to this file

      --modules string       List of enabled modules (comma separated)

      --once                 Run filebeat only once until all harvesters reach EOF

      --path.config string   Configuration path

      --path.data string     Data path

      --path.home string     Home path

      --path.logs string     Logs path

      --setup                Load sample Kibana dashboards and setup Machine Learning

      --strict.perms         Strict permission checking on config files (default true)

  -v, --v                    Log at INFO level


Use "filebeat [command] --help" for more information about a command.


filebeat.yml 예제)

filebeat:

  prospectors:

    -

      paths:

        - /mnt/apps/apache-tomcat/logs/catalina.out

      encoding: utf-8

      input_type: log


      exclude_lines: ['DEBUG', 'INFO']

      include_lines: ['(E|e)rror', 'ERROR', '(E|e)xception', 'EXCEPTION']


      document_type: error-log


      ignore_older: 5m


      scan_frequency: 10s


      multiline:

        pattern: '^[[:space:]]+|^Caused by:'

        negate: false

        match: after

        max_lines: 10

        timeout: 5s


      tail_files: true


      backoff: 1s

      max_backoff: 10s

      backoff_factor: 2


  registry_file: /mnt/config/filebeat/.filebeat-tomcat


output:

  logstash:

    hosts: ["localhost:5044"]


shipper:


logging:

  to_syslog: false

  to_files: true


  level: warning


  files:

    path: /mnt/logs/filebeat

    name: filebeat-tomcat.log

    rotateeverybytes: 10485760 # = 10MB


logstash 예제)

input {

  beats {

    port => 5044

  }

}


output {

  stdout { codec => rubydebug }


  http {

    url => "https://slack.com/api/chat.postMessage"

    content_type => "application/json"

    http_method => "post"

    format => "json"

    mapping => [ "channel", "slack-bot", "text", "%{message}" ]

    headers => ["Authorization", "Bearer xoxb-XXXXXXXXXXXXXXXXXXXXX"]

  }

#  elasticsearch {

#    host => "localhost"

#    port => "9200"

#    protocol => "http"

#    index => "%{[@metadata][index]}"

#    document_type => "%{[@metadata][type]}"

#  }

}


위 예제는 그냥 filebeat 이 log file 을 리스닝 하고 있다가 error 또는 exception 발생 시 바로 slack 으로 해당 에러는 보내주는 예제 입니다.

:

[Logstash] http output plugin - slack chat

Elastic/Logstash 2018. 7. 9. 12:19


예전에는 output slack_chat 이라고 플러그인을 만들어서 사용을 했었는데 logstash output http 가 있어서 그냥 이걸로 바로 사용하겠습니다.


공식문서에 잘못된 정보가 있기 때문에 그대로 보고 따라 하시면 시간을 낭비 하실 수 있습니다.

(6.3.0 에서 테스트 되었습니다.)


Reference)

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html


input {

stdin {}

}


output {

stdout { codec => rubydebug }

http {

url => "https://slack.com/api/chat.postMessage"

content_type => "application/json"

http_method => "post"

format => "json"

mapping => [ "channel", "CXXXXXXXXX", "text", "%{message}" ]

headers => ["Authorization", "Bearer xoxb-xxxxxxxxxxx"]

}

}


문서 내 잘못된 정보)

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html#plugins-outputs-http-mapping


mapping 이 hash 로 되어 있는데 실제 나와 있는 예제로는 동작 하지 않습니다.


잘못된 예)

   mapping => {"foo" => "%{host}"
              "bar" => "%{type}"}


바른 예)

mapping => [ "channel", "CXXXXXXXXX", "text", "%{message}" ]



:

[Logstash] Logstash 를 이용한 CSV 파일 Import를 하려면

Elastic/Logstash 2018. 4. 24. 11:14

Elastic 사의 공식 문서를 보시면 쉽게 하실 수 있습니다.


기본 flow 는 아래와 같습니다.


CSV -> logstash input file -> Logstash filter csv -> logstash output elasticsearch


각각에 필요한 참조문서는

[Logstash Input File]

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html


[Logstash Filter CSV]

https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html


[Logstash Output Elasticsearch]

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html


[Elasticsearch Indices Templates]

https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html


Template 이 필요한 이유는 csv 파일 데이터에 대한 dynamic mapping 시 의도치 않은 데이터에 대한 형변환 오류를 방지 하기 위함 입니다.

사전에 꼭 정의 하셔서 reindexing 하는 일이 없도록 주의 하시면 좋을 것 같습니다.


:

[Logstash] input file start_position => "end"

Elastic/Logstash 2017. 8. 17. 11:20

먼저 앞서 기술한 input file 에 대한 내용을 먼저 읽어 보시면 이해 하시는데 도움이 됩니다.


※ [Logstash] input file plugin 에 대해서 알아 봅니다.


이전 글은 데이터 유실 방지를 위한 설정과 input file 의 주요 설정 정보에 대해서 알아 봤습니다.

이번 글에서는 반대로 start_position => "end" 로 했을 때 왜 데이터가 유실 되는지 간략하게 살펴 보겠습니다.


설정)

input {

  file {

    path => "/xxxx/logs/test-file.log"

    start_position => "end"

    stat_interval => 1

  }


  file {

    path => "/xxxx/logs/test-file1.log"

    start_position => "end"

    stat_interval => 10

  }

}


output {

    stdout {

      codec => "rubydebug"

    }

}


첫 번째 실행)

$ bin/logstash -f config/test-file.conf


첫번째 실행 후 sincedb)

189766986 1 4 3675


두 번째 실행)

$ bin/logstash -f config/test-file.conf


두번째 실행 후 sincedb)

189766986 1 4 4065


보시는 것 처럼 start_position => "end"로 했을 경우 해당 파일의 end byte offset 정보를 기록하게 됩니다.

이후 sincedb  정보는 변경이 되지 않게 됩니다.

logstash 를 중지 하고 재실행 합니다.

그 동안 test-file.log 에는 계속 데이터가 누적 되도록 하였습니다.

두 번째 실행 된 후 sincedb 값을 확인해 보면 변경 되어 있는 것을 볼 수 있습니다.


이와 같이 첫 번째 offset 정보와 두 번째 offset 정보의 차이 만큼 데이터가 유실 되게 되는 것입니다.


: