[Redis] Master/Slave, Sentinel 구성 부터 Jedis 예제까지.

필요해서 구성해본 내용을 정리 합니다.

복습 차원에서 기록 하는 것이라 별 도움이 안될수 있습니다.

Redis 공식 사이트)

http://redis.io/

Redis 다운로드)

http://redis.io/download

Redis 설치 on OSX)

# 다운로드 받은 후 tar 압축 해제 후 make 했습니다.

$ make

    LINK redis-check-dump

    CC redis-check-aof.o

    LINK redis-check-aof

Hint: It's a good idea to run 'make test' ;)

# 필요 하신 경우 make install 하시면 됩니다.

Redis Configure 참조)

http://redis.io/topics/config

- maxmemory 설정은 32bits 운영체제에서 최대 3GB 이며, 64bits 에서는 리소스 만큼 가능 하다고 합니다.

- expire 설정을 해주는 것 보다 그냥 LRU 설정을 사용하라고 합니다. (근데 이건 서비스 특성과 성능 테스트 후 결정 하는게 좋아 보입니다.)

- redis LRU 알고리즘이 완벽하지 않다(Redis LRU algorithm is not an exact implementation.)는 공식 문서의 설명이 있습니다. 그래서 maxmemory-samples 라는 설정을 하는 것 같습니다.

(http://redis.io/topics/lru-cache)

- 단순 휘발성 데이터에 대한 cache 용도 일때는 rdb, append 모드를 disable 하는게 좋습니다.

- sentinel 설정은 master, slave 구성 시 failover 지원을 하기 위함 입니다. (client 에서는 JedisSentinelPool 을 사용하세요.)

- master election 시 민주적으로 하기 위해 quorum 구성을 합니다.

Redis Master 설정)

daemonize yes

pidfile /redis-server/master/pidfile/redis.pid

port 6379

tcp-backlog 511

bind 127.0.0.1

loglevel debug

logfile /redis-server/master/log/redis.log

maxmemory 64MB

maxmemory-policy allkeys-lru

maxmemory-samples 5

appendonly no

save ""

requirepass redispassword

Redis Slave 설정)

daemonize yes

pidfile /redis-server/slave/pidfile/redis.pid

port 6479

tcp-backlog 511

bind 127.0.0.1

loglevel debug

logfile /redis-server/slave/log/redis.log

maxmemory 64MB

maxmemory-policy allkeys-lru

maxmemory-samples 5

appendonly no

save ""

slaveof 127.0.0.1 6379

masterauth redispassword

Redis Sentinel Quorum 설정)

Sentinel 1)

port 26379

dir "/tmp"

sentinel monitor redis-master 127.0.0.1 6379 2

sentinel auth-pass redis-master redispassword

sentinel down-after-milliseconds redis-master 30000

sentinel parallel-syncs redis-master 1

daemonize yes

pidfile "/redis-server/sentinel1/pidfile/sentinel.pid"

logfile "/redis-server/sentinel1/log/sentinel.log"

Sentinel 2)

port 26479

dir "/tmp"

sentinel monitor redis-master 127.0.0.1 6379 2

sentinel auth-pass redis-master redispassword

sentinel down-after-milliseconds redis-master 30000

sentinel parallel-syncs redis-master 1

daemonize yes

pidfile "/redis-server/sentinel2/pidfile/sentinel.pid"

logfile "/redis-server/sentinel2/log/sentinel.log"

Sentinel 3)

port 26579

dir "/tmp"

sentinel monitor redis-master 127.0.0.1 6379 2

sentinel auth-pass redis-master redispassword

sentinel down-after-milliseconds redis-master 30000

sentinel parallel-syncs redis-master 1

daemonize yes

pidfile "/redis-server/sentinel3/pidfile/sentinel.pid"

logfile "/redis-server/sentinel3/log/sentinel.log"

Redis Master/Slave/Sentinel 실행)

$ src/redis-server /redis-server/master/conf/redis.conf

$ src/redis-server /redis-server/slave/conf/redis.conf

$ src/redis-server /redis-server/sentinel1/conf/sentinel.conf --sentinel

$ src/redis-server /redis-server/sentinel2/conf/sentinel.conf --sentinel

$ src/redis-server /redis-server/sentinel3/conf/sentinel.conf --sentinel

Jedis 공식 사이트)

https://github.com/xetorthio/jedis

Jedis 샘플 코드)

- JedisSentinelPoolTest 참조

import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;

import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.junit.Before;
import org.junit.Test;

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisSentinelPool;
import redis.clients.jedis.Transaction;
import redis.clients.jedis.exceptions.JedisConnectionException;
import redis.clients.jedis.exceptions.JedisException;
import redis.clients.jedis.tests.utils.JedisSentinelTestUtil;

public class JedisSentinelPoolTest extends JedisTestBase {
  private static final String MASTER_NAME = "mymaster";

  protected static HostAndPort master = HostAndPortUtil.getRedisServers().get(2);
  protected static HostAndPort slave1 = HostAndPortUtil.getRedisServers().get(3);

  protected static HostAndPort sentinel1 = HostAndPortUtil.getSentinelServers().get(1);
  protected static HostAndPort sentinel2 = HostAndPortUtil.getSentinelServers().get(3);

  protected static Jedis sentinelJedis1;
  protected static Jedis sentinelJedis2;

  protected Set<String> sentinels = new HashSet<String>();

  @Before
  public void setUp() throws Exception {
    sentinels.add(sentinel1.toString());
    sentinels.add(sentinel2.toString());

    sentinelJedis1 = new Jedis(sentinel1.getHost(), sentinel1.getPort());
    sentinelJedis2 = new Jedis(sentinel2.getHost(), sentinel2.getPort());
  }

  @Test(expected = JedisConnectionException.class)
  public void initializeWithNotAvailableSentinelsShouldThrowException() {
    Set<String> wrongSentinels = new HashSet<String>();
    wrongSentinels.add(new HostAndPort("localhost", 65432).toString());
    wrongSentinels.add(new HostAndPort("localhost", 65431).toString());

    JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, wrongSentinels);
    pool.destroy();
  }

  @Test(expected = JedisException.class)
  public void initializeWithNotMonitoredMasterNameShouldThrowException() {
    final String wrongMasterName = "wrongMasterName";
    JedisSentinelPool pool = new JedisSentinelPool(wrongMasterName, sentinels);
    pool.destroy();
  }

  @Test
  public void checkCloseableConnections() throws Exception {
    GenericObjectPoolConfig config = new GenericObjectPoolConfig();

    JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, sentinels, config, 1000,
        "foobared", 2);
    Jedis jedis = pool.getResource();
    jedis.auth("foobared");
    jedis.set("foo", "bar");
    assertEquals("bar", jedis.get("foo"));
    pool.returnResource(jedis);
    pool.close();
    assertTrue(pool.isClosed());
  }

  @Test
  public void ensureSafeTwiceFailover() throws InterruptedException {
    JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, sentinels,
        new GenericObjectPoolConfig(), 2000, "foobared", 2);

    forceFailover(pool);
    // after failover sentinel needs a bit of time to stabilize before a new
    // failover
    Thread.sleep(100);
    forceFailover(pool);

    // you can test failover as much as possible
  }

  @Test
  public void returningBorrowedInstanceBeforeFailoverShouldNotAffectBorrowing()
      throws InterruptedException {
    final JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, sentinels,
        new GenericObjectPoolConfig(), 2000, "foobared", 2);

    Jedis borrowed = pool.getResource();
    forceFailover(pool);

    Thread.sleep(1000);

    // returns instance which was borrowed before failover
    borrowed.close();

    final AtomicBoolean isBorrowed = new AtomicBoolean(false);

    Thread t = new Thread(new Runnable() {
      @Override
      public void run() {
        pool.getResource();
        isBorrowed.set(true);
      }
    });
    t.start();

    // wait for 5 secs
    t.join(5000);

    assertTrue(isBorrowed.get());
  }

  @Test
  public void returnResourceShouldResetState() {
    GenericObjectPoolConfig config = new GenericObjectPoolConfig();
    config.setMaxTotal(1);
    config.setBlockWhenExhausted(false);
    JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, sentinels, config, 1000,
        "foobared", 2);

    Jedis jedis = pool.getResource();
    Jedis jedis2 = null;

    try {
      jedis.set("hello", "jedis");
      Transaction t = jedis.multi();
      t.set("hello", "world");
      pool.returnResource(jedis);

      jedis2 = pool.getResource();

      assertTrue(jedis == jedis2);
      assertEquals("jedis", jedis2.get("hello"));
    } catch (JedisConnectionException e) {
      if (jedis2 != null) {
        pool.returnBrokenResource(jedis2);
        jedis2 = null;
      }
    } finally {
      if (jedis2 != null) jedis2.close();

      pool.destroy();
    }
  }

  @Test
  public void checkResourceIsCloseable() {
    GenericObjectPoolConfig config = new GenericObjectPoolConfig();
    config.setMaxTotal(1);
    config.setBlockWhenExhausted(false);
    JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, sentinels, config, 1000,
        "foobared", 2);

    Jedis jedis = pool.getResource();
    try {
      jedis.set("hello", "jedis");
    } finally {
      jedis.close();
    }

    Jedis jedis2 = pool.getResource();
    try {
      assertEquals(jedis, jedis2);
    } finally {
      jedis2.close();
    }
  }

  @Test
  public void customClientName() {
    GenericObjectPoolConfig config = new GenericObjectPoolConfig();
    config.setMaxTotal(1);
    config.setBlockWhenExhausted(false);
    JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, sentinels, config, 1000,
        "foobared", 0, "my_shiny_client_name");

    Jedis jedis = pool.getResource();

    try {
      assertEquals("my_shiny_client_name", jedis.clientGetname());
    } finally {
      jedis.close();
      pool.destroy();
    }

    assertTrue(pool.isClosed());
  }

  private void forceFailover(JedisSentinelPool pool) throws InterruptedException {
    HostAndPort oldMaster = pool.getCurrentHostMaster();

    // jedis connection should be master
    Jedis beforeFailoverJedis = pool.getResource();
    assertEquals("PONG", beforeFailoverJedis.ping());

    waitForFailover(pool, oldMaster);

    Jedis afterFailoverJedis = pool.getResource();
    assertEquals("PONG", afterFailoverJedis.ping());
    assertEquals("foobared", afterFailoverJedis.configGet("requirepass").get(1));
    assertEquals(2, afterFailoverJedis.getDB().intValue());

    // returning both connections to the pool should not throw
    beforeFailoverJedis.close();
    afterFailoverJedis.close();
  }

  private void waitForFailover(JedisSentinelPool pool, HostAndPort oldMaster)
      throws InterruptedException {
    HostAndPort newMaster = JedisSentinelTestUtil.waitForNewPromotedMaster(MASTER_NAME,
      sentinelJedis1, sentinelJedis2);

    waitForJedisSentinelPoolRecognizeNewMaster(pool, newMaster);
  }

  private void waitForJedisSentinelPoolRecognizeNewMaster(JedisSentinelPool pool,
      HostAndPort newMaster) throws InterruptedException {

    while (true) {
      HostAndPort currentHostMaster = pool.getCurrentHostMaster();

      if (newMaster.equals(currentHostMaster)) break;

      System.out.println("JedisSentinelPool's master is not yet changed, sleep...");

      Thread.sleep(100);
    }
  }

}