jjeong

참고 문서)

https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html

setup.ilm.check_exists

When set to false, disables the check for an existing lifecycle policy.

The default is true.

You need to disable this check if the Filebeat user connecting to a secured cluster doesn’t have the read_ilm privilege.

If you set this option to false, set setup.ilm.overwrite: true so the lifecycle policy can be installed.

setup.ilm.overwrite

When set to true, the lifecycle policy is overwritten at startup. The default is false.

이 설정은 template  설정과 output.elasticsearch.index 랑도 연관이 됩니다.

아주 삽질을 하게 만드는 설정이 될 수도 있습니다.

단, 알고 쓰면 삽질 안하고 모르고 쓰면 삽질 할 수도 있습니다.

늘 그렇지만, Elastic 사는 그냥 모르면 기본 설정만 사용하세요. :)

설정 하고 왜 바로 동작 안하지 그랬는데, 다 이유가 있었습니다.

[LifecycleSettings.java]

/*
 * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
 * or more contributor license agreements. Licensed under the Elastic License;
 * you may not use this file except in compliance with the Elastic License.
 */
package org.elasticsearch.xpack.core.ilm;

import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Setting;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.xpack.core.scheduler.CronSchedule;

/**
 * Class encapsulating settings related to Index Lifecycle Management X-Pack Plugin
 */
public class LifecycleSettings {
    public static final String LIFECYCLE_POLL_INTERVAL = "indices.lifecycle.poll_interval";
    public static final String LIFECYCLE_NAME = "index.lifecycle.name";
    public static final String LIFECYCLE_INDEXING_COMPLETE = "index.lifecycle.indexing_complete";
    public static final String LIFECYCLE_ORIGINATION_DATE = "index.lifecycle.origination_date";
    public static final String LIFECYCLE_PARSE_ORIGINATION_DATE = "index.lifecycle.parse_origination_date";
    public static final String LIFECYCLE_HISTORY_INDEX_ENABLED = "indices.lifecycle.history_index_enabled";

    public static final String SLM_HISTORY_INDEX_ENABLED = "slm.history_index_enabled";
    public static final String SLM_RETENTION_SCHEDULE = "slm.retention_schedule";
    public static final String SLM_RETENTION_DURATION = "slm.retention_duration";


    public static final Setting<TimeValue> LIFECYCLE_POLL_INTERVAL_SETTING = Setting.positiveTimeSetting(LIFECYCLE_POLL_INTERVAL,
        TimeValue.timeValueMinutes(10), Setting.Property.Dynamic, Setting.Property.NodeScope);
    public static final Setting<String> LIFECYCLE_NAME_SETTING = Setting.simpleString(LIFECYCLE_NAME,
        Setting.Property.Dynamic, Setting.Property.IndexScope);
    public static final Setting<Boolean> LIFECYCLE_INDEXING_COMPLETE_SETTING = Setting.boolSetting(LIFECYCLE_INDEXING_COMPLETE, false,
        Setting.Property.Dynamic, Setting.Property.IndexScope);
    public static final Setting<Long> LIFECYCLE_ORIGINATION_DATE_SETTING =
        Setting.longSetting(LIFECYCLE_ORIGINATION_DATE, -1, -1, Setting.Property.Dynamic, Setting.Property.IndexScope);
    public static final Setting<Boolean> LIFECYCLE_PARSE_ORIGINATION_DATE_SETTING = Setting.boolSetting(LIFECYCLE_PARSE_ORIGINATION_DATE,
        false, Setting.Property.Dynamic, Setting.Property.IndexScope);
    public static final Setting<Boolean> LIFECYCLE_HISTORY_INDEX_ENABLED_SETTING = Setting.boolSetting(LIFECYCLE_HISTORY_INDEX_ENABLED,
        true, Setting.Property.NodeScope);


    public static final Setting<Boolean> SLM_HISTORY_INDEX_ENABLED_SETTING = Setting.boolSetting(SLM_HISTORY_INDEX_ENABLED, true,
        Setting.Property.NodeScope);
    public static final Setting<String> SLM_RETENTION_SCHEDULE_SETTING = Setting.simpleString(SLM_RETENTION_SCHEDULE,
        // Default to 1:30am every day
        "0 30 1 * * ?",
        str -> {
        try {
            if (Strings.hasText(str)) {
                // Test that the setting is a valid cron syntax
                new CronSchedule(str);
            }
        } catch (Exception e) {
            throw new IllegalArgumentException("invalid cron expression [" + str + "] for SLM retention schedule [" +
                SLM_RETENTION_SCHEDULE + "]", e);
        }
    }, Setting.Property.Dynamic, Setting.Property.NodeScope);
    public static final Setting<TimeValue> SLM_RETENTION_DURATION_SETTING = Setting.timeSetting(SLM_RETENTION_DURATION,
        TimeValue.timeValueHours(1), TimeValue.timeValueMillis(500), Setting.Property.Dynamic, Setting.Property.NodeScope);
}

설정을 하셨다면 이제는 기다리시면 됩니다. 

Elastic Stack 을 사용하다 보면 Index 를 자동으로 생성 시켜 주는 설정들을 사용 하게 됩니다.

beats 에서는 output elasticsearch 사용 시 설정을 하게 되고,

logstash 에서도 output elasticsearch 사용 시 설정을 하게 됩니다.

보통 이 설정을 아래와 같이 하는데요.

Logstash)

index

- Value type is string
- Default value is "logstash-%{+YYYY.MM.dd}"

Beats)

index

The index name to write events to when you’re using daily indices. 
The default is "beat-%{[agent.version]}-%{+yyyy.MM.dd}" 
(for example, "beat-7.6.2-2020-04-02"). 
If you change this setting, you also need to configure the setup.
template.name and setup.template.pattern options (see Elasticsearch index template).

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "beats-%{[agent.version]}-%{+yyyy.MM.dd}"

- filebeat, metricbeat ...

보시면 아시겠지만, Daily rolling 으로 많이 사용하게 됩니다.

무조건 하루 단위로 생성이 되기 때문에 규모가 너무 작거나 너무 크거나 할 경우 효율이 떨어 질 수 밖에 없습니다.

그래서 Index Lifecycle Management 라는 기능을 Elasticsearch 에서 제공하고 있으니 활용 하시기 바랍니다.

여기서 문제)

그럼 output.elasticsearch.index 설정과 ilm 설정이 동시에 사용 중일 때 어떤 설정에 맞춰서 index 가 생성이 될까요?

정답)

ilm 설정이 우선 합니다. 

Elastic 의 공식 메시지는 이렇습니다.

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html

You can configure index lifecycle management (ILM) policies to automatically manage indices according to your performance, resiliency, and retention requirements.