'ITWeb/개발일반'에 해당되는 글 356건

  1. 2018.02.13 [Script] uniq string line print on shell
  2. 2018.01.12 [git] git log stats 보기
  3. 2017.12.22 [GC] 관련 링크 모음
  4. 2017.12.12 [Java] Executors 로 간단 multithread 테스트
  5. 2017.11.29 [Java] JVM monitoring 도구
  6. 2017.11.24 [Java] Excutors 에서 제공하는 ExecutorService
  7. 2017.11.24 [Java] java.util.concurrent 관련 예제
  8. 2017.11.22 [Bash] for loop range and if condition
  9. 2017.11.18 [Spark] Spark installation on osx
  10. 2017.11.02 [Gradle] Dynamic version cache 설정하기

[Script] uniq string line print on shell

ITWeb/개발일반 2018.02.13 13:26

파일에 id 목록이 들어 있고 여기서 unique 한 id 들만 출력하고 싶을 경우 아래와 같이 사용 하시면 편하게 찾으 실 수 있습니다.


Tips)

$ sort id_list.log | uniq


Ref)

http://www.tutorialspoint.com/unix_commands/sort.htm

http://www.tutorialspoint.com/unix_commands/uniq.htm

tags : script, Shell, sort, UNIQ
Trackback 0 : Comment 0

[git] git log stats 보기

ITWeb/개발일반 2018.01.12 15:45

그냥 제가 보고 싶은 형태로 구성하기 위해 스크립트 만들어 봤습니다.

참고문서)

https://git-scm.com/docs/git-log

https://git-scm.com/docs/pretty-formats


#!/usr/bin/env bash


DAYS_AGO=$1

DATE=`date -v -"$DAYS_AGO"d +%F`

PWD=`pwd`


echo "Date : $DATE" > "git-stats-$DATE.log"

# This script must exist under git repository.


echo "[Commit Statistics]" >> "git-stats-$DATE.log"

# total commit stats

USER_COUNT=`git log --before="$DATE 23:59:59" --after="$DATE 00:00:00" --format='%aN %ce' | sort -u | wc -l`

echo "Total Commit Users : $USER_COUNT" >> "git-stats-$DATE.log"

git log --shortstat --before="$DATE 23:59:59" --after="$DATE 00:00:00" | grep -E "fil(e|es) changed" | awk '{files+=$1; inserted+=$4; deleted+=$6} END {print "files changed: ", files, "lines inserted: ", inserted, "lines deleted: ", deleted }' >> "git-stats-$DATE.log"

echo "" >> "git-stats-$DATE.log"


echo "[Committers]" >> "git-stats-$DATE.log"

# commit user list

git log --before="$DATE 23:59:59" --after="$DATE 00:00:00" --format='%aN %ce' | sort -u >> "git-stats-$DATE.log"

echo "" >> "git-stats-$DATE.log"


echo "[Commit Logs]" >> "git-stats-$DATE.log"

# each user commit stats

git log --before="$DATE 23:59:59" --after="$DATE 00:00:00" --all --pretty=format:'"%an", "%aD", "%s",' --shortstat --no-merges >> "git-stats-$DATE.log"


tags : GIT, Log, stats
Trackback 0 : Comment 0

[GC] 관련 링크 모음

ITWeb/개발일반 2017.12.22 12:44

[GC tuning 글 모음]

https://blogs.apache.org/hbase/entry/tuning_g1gc_for_your_hbase


https://logonjava.blogspot.kr/2015/08/java-g1-gc-full-gc.html

http://product.hubspot.com/blog/g1gc-fundamentals-lessons-from-taming-garbage-collection

http://xmlandmore.blogspot.com/2014/09/g1-gc-tuning-mixed-garbage-collections.html


https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/

http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html

http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html#G1Options


http://d2.naver.com/helloworld/1329

http://d2.naver.com/helloworld/4717

http://d2.naver.com/helloworld/6043

http://d2.naver.com/helloworld/37111

http://d2.naver.com/helloworld/329631


tags : g1gc, gc, java, JVM, tuning
Trackback 0 : Comment 0

[Java] Executors 로 간단 multithread 테스트

ITWeb/개발일반 2017.12.12 11:54

이전 글의 참고 문서들을 먼저 보시면 좋습니다.

http://jjeong.tistory.com/1296

package hello.executors;

import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;

public class HelloExecutors {

public static class HelloCallableThread implements Callable<Integer> {
int input;

public HelloCallableThread(int input) {
this.input = input;
}

@Override
public Integer call() throws Exception {
return input + 1;
}
}

public static void main(String[] args) throws Exception {
ExecutorService executorService = Executors.newFixedThreadPool(4);
Set<Callable<Integer>> callables = new HashSet<Callable<Integer>>();

callables.add(new HelloCallableThread(1));
callables.add(new HelloCallableThread(2));
callables.add(new HelloCallableThread(3));
callables.add(new HelloCallableThread(4));

List<Future<Integer>> futures = executorService.invokeAll(callables);

for(Future<Integer> future : futures){
System.out.println("future.get = " + future.get());
}

executorService.shutdown();
executorService.awaitTermination(1, TimeUnit.SECONDS);
}
}


Trackback 0 : Comment 0

[Java] JVM monitoring 도구

ITWeb/개발일반 2017.11.29 10:33


JMX 를 통한 JVM monitoring 도구

$ jvisualvm

실행 후 remote connection 을 하시고 접속 정보는 접속할 host 명과 JMX port 를 넣으시면 됩니다.

Tools -> Plugins 를 선택 하셔서 필요한 plugin 을 몽땅 설치 하시면 편리 합니다.

tags : JMX, jvisualvm, JVM
Trackback 0 : Comment 0

[Java] Excutors 에서 제공하는 ExecutorService

ITWeb/개발일반 2017.11.24 15:36

구글링 하기 귀찮아서 소소 코드에 있는 주석이랑 코드 가져 왔습니다.

/**
* Creates a thread pool that reuses a fixed number of threads
* operating off a shared unbounded queue. At any point, at most
* {@code nThreads} threads will be active processing tasks.
* If additional tasks are submitted when all threads are active,
* they will wait in the queue until a thread is available.
* If any thread terminates due to a failure during execution
* prior to shutdown, a new one will take its place if needed to
* execute subsequent tasks. The threads in the pool will exist
* until it is explicitly {@link ExecutorService#shutdown shutdown}.
*
* @param nThreads the number of threads in the pool
* @return the newly created thread pool
* @throws IllegalArgumentException if {@code nThreads <= 0}
*/
public static ExecutorService newFixedThreadPool(int nThreads) {
return new ThreadPoolExecutor(nThreads, nThreads,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>());
}
/**
* Creates a thread pool that maintains enough threads to support
* the given parallelism level, and may use multiple queues to
* reduce contention. The parallelism level corresponds to the
* maximum number of threads actively engaged in, or available to
* engage in, task processing. The actual number of threads may
* grow and shrink dynamically. A work-stealing pool makes no
* guarantees about the order in which submitted tasks are
* executed.
*
* @param parallelism the targeted parallelism level
* @return the newly created thread pool
* @throws IllegalArgumentException if {@code parallelism <= 0}
* @since 1.8
*/
public static ExecutorService newWorkStealingPool(int parallelism) {
return new ForkJoinPool
(parallelism,
ForkJoinPool.defaultForkJoinWorkerThreadFactory,
null, true);
}
/**
* Creates a work-stealing thread pool using all
* {@link Runtime#availableProcessors available processors}
* as its target parallelism level.
* @return the newly created thread pool
* @see #newWorkStealingPool(int)
* @since 1.8
*/
public static ExecutorService newWorkStealingPool() {
return new ForkJoinPool
(Runtime.getRuntime().availableProcessors(),
ForkJoinPool.defaultForkJoinWorkerThreadFactory,
null, true);
}
/**
* Creates a thread pool that reuses a fixed number of threads
* operating off a shared unbounded queue, using the provided
* ThreadFactory to create new threads when needed. At any point,
* at most {@code nThreads} threads will be active processing
* tasks. If additional tasks are submitted when all threads are
* active, they will wait in the queue until a thread is
* available. If any thread terminates due to a failure during
* execution prior to shutdown, a new one will take its place if
* needed to execute subsequent tasks. The threads in the pool will
* exist until it is explicitly {@link ExecutorService#shutdown
* shutdown}.
*
* @param nThreads the number of threads in the pool
* @param threadFactory the factory to use when creating new threads
* @return the newly created thread pool
* @throws NullPointerException if threadFactory is null
* @throws IllegalArgumentException if {@code nThreads <= 0}
*/
public static ExecutorService newFixedThreadPool(int nThreads, ThreadFactory threadFactory) {
return new ThreadPoolExecutor(nThreads, nThreads,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(),
threadFactory);
}
/**
* Creates an Executor that uses a single worker thread operating
* off an unbounded queue. (Note however that if this single
* thread terminates due to a failure during execution prior to
* shutdown, a new one will take its place if needed to execute
* subsequent tasks.) Tasks are guaranteed to execute
* sequentially, and no more than one task will be active at any
* given time. Unlike the otherwise equivalent
* {@code newFixedThreadPool(1)} the returned executor is
* guaranteed not to be reconfigurable to use additional threads.
*
* @return the newly created single-threaded Executor
*/
public static ExecutorService newSingleThreadExecutor() {
return new FinalizableDelegatedExecutorService
(new ThreadPoolExecutor(1, 1,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>()));
}
/**
* Creates an Executor that uses a single worker thread operating
* off an unbounded queue, and uses the provided ThreadFactory to
* create a new thread when needed. Unlike the otherwise
* equivalent {@code newFixedThreadPool(1, threadFactory)} the
* returned executor is guaranteed not to be reconfigurable to use
* additional threads.
*
* @param threadFactory the factory to use when creating new
* threads
*
* @return the newly created single-threaded Executor
* @throws NullPointerException if threadFactory is null
*/
public static ExecutorService newSingleThreadExecutor(ThreadFactory threadFactory) {
return new FinalizableDelegatedExecutorService
(new ThreadPoolExecutor(1, 1,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(),
threadFactory));
}


/**
* Creates a thread pool that creates new threads as needed, but
* will reuse previously constructed threads when they are
* available. These pools will typically improve the performance
* of programs that execute many short-lived asynchronous tasks.
* Calls to {@code execute} will reuse previously constructed
* threads if available. If no existing thread is available, a new
* thread will be created and added to the pool. Threads that have
* not been used for sixty seconds are terminated and removed from
* the cache. Thus, a pool that remains idle for long enough will
* not consume any resources. Note that pools with similar
* properties but different details (for example, timeout parameters)
* may be created using {@link ThreadPoolExecutor} constructors.
*
* @return the newly created thread pool
*/
public static ExecutorService newCachedThreadPool() {
return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS,
new SynchronousQueue<Runnable>());
}
/**
* Creates a thread pool that creates new threads as needed, but
* will reuse previously constructed threads when they are
* available, and uses the provided
* ThreadFactory to create new threads when needed.
* @param threadFactory the factory to use when creating new threads
* @return the newly created thread pool
* @throws NullPointerException if threadFactory is null
*/
public static ExecutorService newCachedThreadPool(ThreadFactory threadFactory) {
return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS,
new SynchronousQueue<Runnable>(),
threadFactory);
}
/**
* Creates a single-threaded executor that can schedule commands
* to run after a given delay, or to execute periodically.
* (Note however that if this single
* thread terminates due to a failure during execution prior to
* shutdown, a new one will take its place if needed to execute
* subsequent tasks.) Tasks are guaranteed to execute
* sequentially, and no more than one task will be active at any
* given time. Unlike the otherwise equivalent
* {@code newScheduledThreadPool(1)} the returned executor is
* guaranteed not to be reconfigurable to use additional threads.
* @return the newly created scheduled executor
*/
public static ScheduledExecutorService newSingleThreadScheduledExecutor() {
return new DelegatedScheduledExecutorService
(new ScheduledThreadPoolExecutor(1));
}
/**
* Creates a single-threaded executor that can schedule commands
* to run after a given delay, or to execute periodically. (Note
* however that if this single thread terminates due to a failure
* during execution prior to shutdown, a new one will take its
* place if needed to execute subsequent tasks.) Tasks are
* guaranteed to execute sequentially, and no more than one task
* will be active at any given time. Unlike the otherwise
* equivalent {@code newScheduledThreadPool(1, threadFactory)}
* the returned executor is guaranteed not to be reconfigurable to
* use additional threads.
* @param threadFactory the factory to use when creating new
* threads
* @return a newly created scheduled executor
* @throws NullPointerException if threadFactory is null
*/
public static ScheduledExecutorService newSingleThreadScheduledExecutor( ThreadFactory threadFactory) {
return new DelegatedScheduledExecutorService
(new ScheduledThreadPoolExecutor(1, threadFactory));
}
/**
* Creates a thread pool that can schedule commands to run after a
* given delay, or to execute periodically.
* @param corePoolSize the number of threads to keep in the pool,
* even if they are idle
* @return a newly created scheduled thread pool
* @throws IllegalArgumentException if {@code corePoolSize < 0}
*/
public static ScheduledExecutorService newScheduledThreadPool(int corePoolSize) {
return new ScheduledThreadPoolExecutor(corePoolSize);
}


/**
* Creates a thread pool that can schedule commands to run after a
* given delay, or to execute periodically.
* @param corePoolSize the number of threads to keep in the pool,
* even if they are idle
* @param threadFactory the factory to use when the executor
* creates a new thread
* @return a newly created scheduled thread pool
* @throws IllegalArgumentException if {@code corePoolSize < 0}
* @throws NullPointerException if threadFactory is null
*/
public static ScheduledExecutorService newScheduledThreadPool(
int corePoolSize, ThreadFactory threadFactory) {
return new ScheduledThreadPoolExecutor(corePoolSize, threadFactory);
}
/**
* Returns an object that delegates all defined {@link
* ExecutorService} methods to the given executor, but not any
* other methods that might otherwise be accessible using
* casts. This provides a way to safely "freeze" configuration and
* disallow tuning of a given concrete implementation.
* @param executor the underlying implementation
* @return an {@code ExecutorService} instance
* @throws NullPointerException if executor null
*/
public static ExecutorService unconfigurableExecutorService(ExecutorService executor) {
if (executor == null)
throw new NullPointerException();
return new DelegatedExecutorService(executor);
}
/**
* Returns an object that delegates all defined {@link
* ScheduledExecutorService} methods to the given executor, but
* not any other methods that might otherwise be accessible using
* casts. This provides a way to safely "freeze" configuration and
* disallow tuning of a given concrete implementation.
* @param executor the underlying implementation
* @return a {@code ScheduledExecutorService} instance
* @throws NullPointerException if executor null
*/
public static ScheduledExecutorService unconfigurableScheduledExecutorService( ScheduledExecutorService executor) {
if (executor == null)
throw new NullPointerException();
return new DelegatedScheduledExecutorService(executor);
}


  • newFixedThreadPool
    • 정해준 크기 만큼의 쓰레드를 생성하고 재사용 합니다. 명시적으로 shutdown() 하지 않는 한 쓰레드 중 하나가 종료 되면 다시 생성을 하게 됩니다.


  • newWorkStealingPool
    • 작업 순서에 대한 보장은 하지 않습니다, parallelism 수준에 따라 쓰레드를 충분히 지원 하지만 다중큐를 사용하는 것이 좋습니다. 쓰레드의 크기는 동적으로 늘었다 줄었다 합니다.


  • newSingleThreadExecutor
    • 쓰레드를 하나만 생성해서 사용합니다. 만약 종료 되면 다시 쓰레드가 생성이 되며 작업에 대한 연속성을 보장해 줍니다.


  • newCachedThreadPool
    • 필요한 만큼 쓰레드를 생성 하게 됩니다. 하지만 60초 동안 사용되지 않으면 풀에서 제거 됩니다.
    • 60초가 기본 설정 값 이며, 생성된 쓰레드는 재사용 됩니다.


  • newSingleThreadScheduledExecutor
    • 스케쥴링이 가능한 하나의 쓰레드를 생성 합니다. 스케쥴 기능을 빼고는 newSingleThreadExecutor 와 비슷 하다고 보시면 됩니다.


  • newScheduledThreadPool
    • 스케쥴링이 가능한 쓰레드 풀을 생성 합니다. 쓰레드가 idle 상태에 있더라도 종료 되거나 소멸 되지 않고 풀에 그대로 남아 있습니다.


Trackback 0 : Comment 0

[Java] java.util.concurrent 관련 예제

ITWeb/개발일반 2017.11.24 15:03

구글링 하기 귀찮을 땐.

http://tutorials.jenkov.com/java-util-concurrent/index.html

http://winterbe.com/posts/2015/04/07/java8-concurrency-tutorial-thread-executor-examples/


그리고 java 8 관련해서는 위에 링크 중 아래 블로그 쥔장이 잘 정리해 둔것 같내요.

http://winterbe.com/java/

Trackback 0 : Comment 0

[Bash] for loop range and if condition

ITWeb/개발일반 2017.11.22 11:16

이런 간단한것도 매번 생각이 나지 않아서 기록해 봅니다.


$ for i in {1..10}

do

s3cmd get s3://part-$i

if [ $i -gt 9 ]

then

break

fi

done


tags : bash, condition, for, IF, loop
Trackback 0 : Comment 0

[Spark] Spark installation on osx

ITWeb/개발일반 2017.11.18 21:54

기본 설치 입니다.

제가 brew install 보다는 직접 binary 받아서 설치 하는걸 더 선호해서 올려 봅니다.

[Spark installation on osx]

Ref. https://isaacchanghau.github.io/2017/06/28/Spark-Installation-on-Mac-OS-X/


1. scala

https://www.scala-lang.org/download/


$ tar -xvzf scala-2.12.4.tgz

$ vi .bash_profile

export SCALA_HOME=/Users/henry/Work/apps/scala-2.12.4

PATH=$SCALA_HOME/bin:$PATH

2. spark

https://spark.apache.org/downloads.html


$ tar -xvzf spark-2.2.0-bin-hadoop2.7.tgz

$ vi .bash_profile

export SPARK_HOME=/Users/henry/Work/apps/spark-2.2.0-bin-hadoop2.7

PATH=$SPARK_HOME/bin:$PATH

$ cd /Users/henry/Work/apps/spark-2.2.0-bin-hadoop2.7

$ cp spark-env.sh.template spark-env.sh

$ vim spark-env.sh

export SCALA_HOME=/Users/henry/Work/apps/scala-2.12.4

export SPARK_MASTER_IP=localhost

export SPARK_WORKER_MEMORY=1g

$ spark-shell


scala 와 spark 이 설치가 되어야 합니다.

보시면 아시겠지만 .bash_profile 에 환경 설정 해주시고 실행하시면 됩니다.

PATH=$SCALA_HOME/bin:$SPARK_HOME/bin:$PATH

$ source .bash_profile

$ spark-shell

Spark context Web UI available at http://127.0.0.1:4040

Spark context available as 'sc' (master = local[*], app id = local-1511009262380).

Spark session available as 'spark'.

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0

      /_/


Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_72)

Type in expressions to have them evaluated.

Type :help for more information.


scala>


tags : install, OSX, Scala, Spark
Trackback 0 : Comment 0

[Gradle] Dynamic version cache 설정하기

ITWeb/개발일반 2017.11.02 21:45

지난 글과 함께 보시면 좋습니다.


[지난 글]

[Gradle] Dependency filename 구하기

[Gradle] Use latest version on dependency jar.


[참고문서]

https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html


[설정]

configurations {

dependencyFn

}


configurations.dependencyFn {

resolutionStrategy.cacheDynamicVersionsFor 10, 'minutes'

}


dependencies {

dependencyFn group: "org.apaceh.lucene.analysis.arirang", name: "arirang-dictionary", \

version: "+", changing: true

}


Trackback 0 : Comment 0