'2020/06'에 해당되는 글 9건

  1. 2020.06.25 [Spring] Spring Cloud Config 설정 파일 적용
  2. 2020.06.25 [Spring] Spring boot tomcat relaxedQueryChars
  3. 2020.06.19 [Elasticsearch] X-pack Security API Key 사용 해 보기
  4. 2020.06.18 [Spring] Spring Cloud Config 다중 Backend 사용 시...
  5. 2020.06.17 [Elasticsearch] script 사용 시 "#! Deprecation: Deprecated field [inline] used, expected [source] instead"
  6. 2020.06.16 [Spring] @RefreshScope 적용 시 반영이 잘 안될 때
  7. 2020.06.03 [Filebeat] ilm 조심 하자.
  8. 2020.06.03 [Filebeat] template 설정 setup.template.append_fields
  9. 2020.06.01 [Spring] Spring boot jar build and run.

[Spring] Spring Cloud Config 설정 파일 적용

ITWeb/개발일반 2020. 6. 25. 14:29

spring config client  에 server config 파일이 존재 할 경우 적용 안되는 문제를 해결 하기 위해 아래 설정을 하면 됩니다.

 

https://cloud.spring.io/spring-cloud-static/spring-cloud-commons/2.2.2.RELEASE/reference/html/#overriding-bootstrap-properties

 

1.4. Overriding the Values of Remote Properties

The property sources that are added to your application by the bootstrap context are often “remote” (from example, from Spring Cloud Config Server). By default, they cannot be overridden locally. If you want to let your applications override the remote properties with their own system properties or config files, the remote property source has to grant it permission by setting spring.cloud.config.allowOverride=true (it does not work to set this locally). Once that flag is set, two finer-grained settings control the location of the remote properties in relation to system properties and the application’s local configuration:

  • spring.cloud.config.overrideNone=true: Override from any local property source.

  • spring.cloud.config.overrideSystemProperties=false: Only system properties, command line arguments, and environment variables (but not the local config files) should override the remote settings.

spring:
  cloud:
    config:
      uri: xxxx
      allowOverride: true
      overrideNone: true
      overrideSystemProperties: false

 

:

[Spring] Spring boot tomcat relaxedQueryChars

ITWeb/개발일반 2020. 6. 25. 11:35

https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#server-properties

 

Common Application properties

Various properties can be specified inside your application.properties file, inside your application.yml file, or as command line switches. This appendix provides a list of common Spring Boot properties and references to the underlying classes that consume

docs.spring.io

https://tomcat.apache.org/tomcat-9.0-doc/config/http.html

 

Apache Tomcat 9 Configuration Reference (9.0.36) - The HTTP Connector

This Connector supports all of the required features of the HTTP/1.1 protocol, as described in RFCs 7230-7235, including persistent connections, pipelining, expectations and chunked encoding. If the client supports only HTTP/1.0 or HTTP/0.9, the Connector

tomcat.apache.org

 

server.tomcat.relaxed-path-chars

Comma-separated list of additional unencoded characters that should be allowed in URI paths. Only "< > [ \ ] ^ ` { | }" are allowed.

server.tomcat.relaxed-query-chars

Comma-separated list of additional unencoded characters that should be allowed in URI query strings. Only "< > [ \ ] ^ ` { | }" are allowed.

relaxedPathChars

The HTTP/1.1 specification requires that certain characters are %nn encoded when used in URI paths. Unfortunately, many user agents including all the major browsers are not compliant with this specification and use these characters in unencoded form. To prevent Tomcat rejecting such requests, this attribute may be used to specify the additional characters to allow. If not specified, no additional characters will be allowed. The value may be any combination of the following characters: " < > [ \ ] ^ ` { | } . Any other characters present in the value will be ignored.

relaxedQueryChars

The HTTP/1.1 specification requires that certain characters are %nn encoded when used in URI query strings. Unfortunately, many user agents including all the major browsers are not compliant with this specification and use these characters in unencoded form. To prevent Tomcat rejecting such requests, this attribute may be used to specify the additional characters to allow. If not specified, no additional characters will be allowed. The value may be any combination of the following characters: " < > [ \ ] ^ ` { | } . Any other characters present in the value will be ignored.

의도치 않게 tomcat 에서 에러 처리를 해버려서 이를 방지 하기 위해 설정 후 application 에서 처리 하시면 됩니다.

:

[Elasticsearch] X-pack Security API Key 사용 해 보기

Elastic/Elasticsearch 2020. 6. 19. 11:07

Elastic Stack 이 좋은 이유는 기본 Basic license 까지 사용이 가능 하다는 것입니다.

사실 이것 말고도 엄청 많죠 ㅎㅎ 

 

https://www.elastic.co/subscriptions

 

딱 API keys management 까지 사용이 됩니다. ㅎㅎㅎ

 

먼저 사용하기에 앞서서 Elasticsearch 와 Kibana 에 x-pack 사용을 위한 설정을 하셔야 합니다.

 

[Elasticsearch]

- elasticsearch.yml

xpack.monitoring.enabled: true
xpack.ml.enabled: true
xpack.security.enabled: true

xpack.security.authc.api_key.enabled: true
xpack.security.authc.api_key.hashing.algorithm: "pbkdf2"
xpack.security.authc.api_key.cache.ttl: "1d"
xpack.security.authc.api_key.cache.max_keys: 10000
xpack.security.authc.api_key.cache.hash_algo: "ssha256"

위 설정은 기본이기 때문에 환경에 맞게 최적화 하셔야 합니다.

https://www.elastic.co/guide/en/elasticsearch/reference/7.8/security-settings.html#api-key-service-settings

 

[Kibana]

- kibana.yml

xpack:
  security:
    enabled: true
    encryptionKey: "9c42bff2e04f9b937966bda03e6b5828"
    session:
      idleTimeout: 600000
    audit:
      enabled: true

 

이렇게 설정 한 후 id/password 설정을 하시면 됩니다.

 

# bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]

Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

 

이렇게 설정이 끝나면 kibana 에 접속해서 API key 를 생성 하시면 됩니다.

아래 문서는 생성 시 도움이 되는 문서 입니다.

 

www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html

www.elastic.co/guide/en/elasticsearch/reference/7.7/security-api-put-role.htmlwww.elastic.co/guide/en/elasticsearch/reference/7.7/defining-roles.htmlwww.elastic.co/guide/en/elasticsearch/reference/7.7/security-api-create-api-key.html

 

Kibana Console 에서 아래와 같이 생성이 가능 합니다.

POST /_security/api_key
{
  "name": "team-index-command",
  "expiration": "10m", 
  "role_descriptors": { 
    "role-team-index-command": {
      "cluster": ["all"],
      "index": [
        {
          "names": ["*"],
          "privileges": ["all"]
        }
      ]
    }
  }
}

{
  "id" : "87cuynIBjKAXtnkobGgo",
  "name" : "team-index-command",
  "expiration" : 1592529999478,
  "api_key" : "OlVGT_Q8RGq1C_ASHW7pGg"
}

생성 이후 사용을 위해서는 

 

- ApiKey 는 id:api_key 를 base64 인코딩 합니다.

base64_encode("87cuynIBjKAXtnkobGgo"+":"+"OlVGT_Q8RGq1C_ASHW7pGg")
==> VGVVOXluSUJHUUdMaHpvcUxDVWo6aUtfSmlEMmdSMy1FUUFpdENCYzF1QQ==
curl -H 
  "Authorization: ApiKey VGVVOXluSUJHUUdMaHpvcUxDVWo6aUtfSmlEMmdSMy1FUUFpdENCYzF1QQ==" 
  http://localhost:9200/_cluster/health

이제 용도와 목적에 맞춰서 API key 를 만들고 사용 하시면 되겠습니다.

 

:

[Spring] Spring Cloud Config 다중 Backend 사용 시...

ITWeb/개발일반 2020. 6. 18. 10:41

Spring Cloud Config 를 이용해서 다중 Backend 를 구성 할 수 있습니다.

그러나 order 옵션이 저는 제대로 동작 하지 않아서 실행 시 사용해야 하는 Backend 에 대한 profile 로 관리하기로 했습니다.

 

동시에 두 개의 Backend 를 사용 하는 것도 가능 한데 이게 같은 설정 파일이 양쪽에 다 존재 할 때 order 에 맞춰서 설정 정보를 가져 와야 하는데 이상하게도 git 에 있는 설정을 먼저 가져 와서 목적에 맞게 사용을 할 수 없었습니다.

 

spring:
  application:
    name: config-server
  profiles:
    active: awss3, git
  cloud:
    config:
      server:
        awss3:
          region: ap-northeast-2
          bucket: ${BUCKET-NAME}
          order: 1
        git:
          uri: https://git/config-repo.git
          skipSslValidation: true
          username: xxxxx
          password: xxxxx
          clone-on-start: true
          order: 2

저렇게 설정 하고 했었는데 잘 못된 부분이 있다면 댓글 좀 달아 주세요. :)

 

profile을 실행 시점에 awss3 나 git 으로 설정해서 사용 하도록 마무리 했습니다.

:

[Elasticsearch] script 사용 시 "#! Deprecation: Deprecated field [inline] used, expected [source] instead"

Elastic/Elasticsearch 2020. 6. 17. 08:20

에러 메시지를 보면 답이 나와 있습니다.

inline 대신 source 를 사용 하라는 이야기 입니다.

 

[ASIS]

  "aggs": {
    "3": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30s",
        "time_zone": "Asia/Seoul",
        "min_doc_count": 1
      },
      "aggs": {
        "1": {
          "max": {
            "field": "system.cpu.total.pct",
            "script": {
              "inline": "doc['system.cpu.total.pct'].value *100",
              "lang": "painless"
            }
          }
        }
      }
    }
  }

 

[TOBE]

  "aggs": {
    "3": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30s",
        "time_zone": "Asia/Seoul",
        "min_doc_count": 1
      },
      "aggs": {
        "1": {
          "max": {
            "field": "system.cpu.total.pct",
            "script": {
              "source": "doc['system.cpu.total.pct'].value *100",
              "lang": "painless"
            }
          }
        }
      }
    }
  }

이상 끝.

:

[Spring] @RefreshScope 적용 시 반영이 잘 안될 때

ITWeb/개발일반 2020. 6. 16. 14:51

일단 spring config server 에서 설정을 변경해서 적용 합니다.

 

POST , /actuator/refresh 를  실행 합니다.

 

그럼 기대 하는 건 변경 된 설정 값이 반영이 되어 있어야 한다는 것입니다.

 

그러나 반영이 안되어 있어서 디버깅 하다 아래와 같이 event 를 잡아서 처리를 해주니 잘되는 것을 확인 했습니다.

 

  @EventListener(RefreshScopeRefreshedEvent.class)
  public void onRefresh() {
    log.debug("triggering ----> {}", watcherThreshold);
  }

참고해서 보셔야 하는 코드는 아래 입니다.

/*
 * Copyright 2012-2020 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      https://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.springframework.cloud.context.scope.refresh;

import java.io.Serializable;

import org.springframework.beans.BeansException;
import org.springframework.beans.factory.config.BeanDefinition;
import org.springframework.beans.factory.support.BeanDefinitionRegistry;
import org.springframework.cloud.context.scope.GenericScope;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.context.ApplicationListener;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.core.Ordered;
import org.springframework.jmx.export.annotation.ManagedOperation;
import org.springframework.jmx.export.annotation.ManagedResource;

/**
 * <p>
 * A Scope implementation that allows for beans to be refreshed dynamically at runtime
 * (see {@link #refresh(String)} and {@link #refreshAll()}). If a bean is refreshed then
 * the next time the bean is accessed (i.e. a method is executed) a new instance is
 * created. All lifecycle methods are applied to the bean instances, so any destruction
 * callbacks that were registered in the bean factory are called when it is refreshed, and
 * then the initialization callbacks are invoked as normal when the new instance is
 * created. A new bean instance is created from the original bean definition, so any
 * externalized content (property placeholders or expressions in string literals) is
 * re-evaluated when it is created.
 * </p>
 *
 * <p>
 * Note that all beans in this scope are <em>only</em> initialized when first accessed, so
 * the scope forces lazy initialization semantics.
 * </p>
 *
 * <p>
 * The scoped proxy approach adopted here has a side benefit that bean instances are
 * automatically {@link Serializable}, and can be sent across the wire as long as the
 * receiver has an identical application context on the other side. To ensure that the two
 * contexts agree that they are identical, they have to have the same serialization ID.
 * One will be generated automatically by default from the bean names, so two contexts
 * with the same bean names are by default able to exchange beans by name. If you need to
 * override the default ID, then provide an explicit {@link #setId(String) id} when the
 * Scope is declared.
 * </p>
 *
 * @author Dave Syer
 * @since 3.1
 *
 */
@ManagedResource
public class RefreshScope extends GenericScope implements ApplicationContextAware,
		ApplicationListener<ContextRefreshedEvent>, Ordered {

	private ApplicationContext context;

	private BeanDefinitionRegistry registry;

	private boolean eager = true;

	private int order = Ordered.LOWEST_PRECEDENCE - 100;

	/**
	 * Creates a scope instance and gives it the default name: "refresh".
	 */
	public RefreshScope() {
		super.setName("refresh");
	}

	@Override
	public int getOrder() {
		return this.order;
	}

	public void setOrder(int order) {
		this.order = order;
	}

	/**
	 * Flag to determine whether all beans in refresh scope should be instantiated eagerly
	 * on startup. Default true.
	 * @param eager The flag to set.
	 */
	public void setEager(boolean eager) {
		this.eager = eager;
	}

	@Override
	public void postProcessBeanDefinitionRegistry(BeanDefinitionRegistry registry)
			throws BeansException {
		this.registry = registry;
		super.postProcessBeanDefinitionRegistry(registry);
	}

	@Override
	public void onApplicationEvent(ContextRefreshedEvent event) {
		start(event);
	}

	public void start(ContextRefreshedEvent event) {
		if (event.getApplicationContext() == this.context && this.eager
				&& this.registry != null) {
			eagerlyInitialize();
		}
	}

	private void eagerlyInitialize() {
		for (String name : this.context.getBeanDefinitionNames()) {
			BeanDefinition definition = this.registry.getBeanDefinition(name);
			if (this.getName().equals(definition.getScope())
					&& !definition.isLazyInit()) {
				Object bean = this.context.getBean(name);
				if (bean != null) {
					bean.getClass();
				}
			}
		}
	}

	@ManagedOperation(description = "Dispose of the current instance of bean name "
			+ "provided and force a refresh on next method execution.")
	public boolean refresh(String name) {
		if (!name.startsWith(SCOPED_TARGET_PREFIX)) {
			// User wants to refresh the bean with this name but that isn't the one in the
			// cache...
			name = SCOPED_TARGET_PREFIX + name;
		}
		// Ensure lifecycle is finished if bean was disposable
		if (super.destroy(name)) {
			this.context.publishEvent(new RefreshScopeRefreshedEvent(name));
			return true;
		}
		return false;
	}

	@ManagedOperation(description = "Dispose of the current instance of all beans "
			+ "in this scope and force a refresh on next method execution.")
	public void refreshAll() {
		super.destroy();
		this.context.publishEvent(new RefreshScopeRefreshedEvent());
	}

	@Override
	public void setApplicationContext(ApplicationContext context) throws BeansException {
		this.context = context;
	}

}

 

refreshAll() 을 호출 하게 되어서 반영이 잘됩니다.

 

더불어 그냥 프레임워크에서 제공하는 @Scheduled 사용 하십시오.

 

  @Scheduled(fixedDelayString = "${megatoi.watcher.cpu.period}000")
  private void runner() {
    megatoiCheckerUtil.setWatcherThreshold(watcherThreshold);
    megatoiCheckerUtil.setPayload(getCheckQuery());
    megatoiCheckerUtil.setResourceType(ResourceType.CPU);
    megatoiCheckerUtil.commonRunner();
  }
:

[Filebeat] ilm 조심 하자.

Elastic/Beats 2020. 6. 3. 14:19

참고 문서)

https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html

 

setup.ilm.check_exists

When set to false, disables the check for an existing lifecycle policy.

The default is true.

You need to disable this check if the Filebeat user connecting to a secured cluster doesn’t have the read_ilm privilege.

If you set this option to false, set setup.ilm.overwrite: true so the lifecycle policy can be installed.

 

setup.ilm.overwrite

When set to true, the lifecycle policy is overwritten at startup. The default is false.

 

이 설정은 template  설정과 output.elasticsearch.index 랑도 연관이 됩니다.

아주 삽질을 하게 만드는 설정이 될 수도 있습니다.

단, 알고 쓰면 삽질 안하고 모르고 쓰면 삽질 할 수도 있습니다.

 

늘 그렇지만, Elastic 사는 그냥 모르면 기본 설정만 사용하세요. :)

:

[Filebeat] template 설정 setup.template.append_fields

Elastic/Beats 2020. 6. 3. 14:15

참고 문서)

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html

 

설정 중에 필요한 내용이 있어서 기록해 봅니다.

 

setup.template.append_fields

 

A list of fields to be added to the template and Kibana index pattern.

This setting adds new fields. It does not overwrite or change existing fields.

This setting is useful when your data contains fields that Filebeat doesn’t know about in advance.

If append_fields is specified along with overwrite: true,

Filebeat overwrites the existing template and applies the new template when creating new indices.

Existing indices are not affected.

If you’re running multiple instances of Filebeat with different append_fields settings,

the last one writing the template takes precedence.

Any changes to this setting also affect the Kibana index pattern.

 

Example config:

setup.template.overwrite: true

setup.template.append_fields:

  - name: test.name

     type: keyword

  - name: test.hostname

     type: long

 

이 설정으로 dynamic mapping 이나 template 관련 번거로움을 간단하게 해결 할 수 있습니다.

저는 몇 개 field 에 대해서 date 로 설정을 해야 해서 이 설정을 사용했습니다.

 

:

[Spring] Spring boot jar build and run.

ITWeb/개발일반 2020. 6. 1. 18:05

build.gradle)
plugins {
  id 'org.springframework.boot' version '2.3.0.RELEASE'
...중략...
  id 'java'
  id 'war'
}

...중략...

task projectVersion {
  doLast {
    println "${project.version}"
  }
}

task jarName {
  doLast {
    println "${archivesBaseName}-${version}.jar"
  }
}

bootJar {
  archivesBaseName = "springboot-was"
}

war {
  enabled = true
  archivesBaseName = "springboot-was"
}

 

build)
$ ./gradlew clean build bootJar

 

run)
$ sudo java -Djava.security.egd=file:/dev/./urandom -jar springboot-was.jar

 

springboot 프로젝트로 빌드 후 embedded tomcat 으로 실행 하는 예제 입니다.

: