jjeong

 java.io.IOException: No FileSystem for scheme: hdfs

Configuration conf = new Configuration();

// conf.set("fs.default.name", hdfsServer); // deprecated

conf.set("fs.defaultFS", hdfsServer);

FileSystem hdfs = FileSystem.get(conf);

BufferedReader br = new BufferedReader(new InputStreamReader(hdfs.open(new Path(schemaFile))));

while ((line = br.readLine()) != null) {

....

}

br.close();

java -classpath /Users/hwjeong/Documents/workspace/eclipse-j2ee/PROJECT/target/standalone_lib -jar /Users/hwjeong/Documents/workspace/eclipse-j2ee/PROJECT/target/standalone.jar StandaloneMain hdfs://localhost:9000

<dependency>

    <groupId>org.apache.hadoop</groupId>

    <artifactId>hadoop-hdfs</artifactId>

    <version>2.3.0</version>

</dependency>

/*

  public static void readLines(Path location, Configuration conf) throws Exception {

    CompressionCodec codec = (CompressionCodec) ReflectionUtils.newInstance(SnappyCodec.class, conf);
    FileSystem hdfs = FileSystem.get(conf);
    
    try{
      FileStatus[] fileStatus = hdfs.listStatus(location);
      
      for (FileStatus file : fileStatus) {
        if ( file.isDirectory() ) {
          continue;
        }
        
        CompressionInputStream cis = codec.createInputStream(hdfs.open(file.getPath()));
        BufferedReader br = new BufferedReader(new InputStreamReader(cis));

        try {
          String line;
          line=br.readLine();
          
          while (line != null){
            line = br.readLine();

                     System.out.println(line);
          }
        } finally {
          br.close();
        }
      }
    } catch ( IOException e ) {
    }
  }

- hadoop-env.sh 에 아래 코드 추가

HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

export HADOOP_OPTS

- eclipse project 에서 run configure 내 arguments 에 코드 추가

-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk

- virtual box

version 4.2.12

http://www.oracle.com/technetwork/server-storage/virtualbox/downloads/index.html

- centos

version 6.4

http://www.centos.org/modules/tinycontent/index.php?id=15

http://ftp.neowiz.com/centos/6.4/isos/x86_64/

- hadoop

version : 1.0.4

http://hadoop.apache.org/releases.html#Download

http://mirror.apache-kr.org/hadoop/common/hadoop-1.0.4/hadoop-1.0.4.tar.gz

- mysql

version : 5.6.11

http://dev.mysql.com/downloads/mysql/

source code 선택

http://dev.mysql.com/get/Downloads/MySQL-5.6/mysql-5.6.11.tar.gz/from/http://cdn.mysql.com/

- hive

version : 0.9.0

http://hive.apache.org/releases.html#Download

http://mirror.apache-kr.org/hive/hive-0.9.0/hive-0.9.0.tar.gz

- haproxy

version : 1.4.23

http://haproxy.1wt.eu/

http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.23.tar.gz

- jdk

version : 1.6.0_45

http://www.oracle.com/technetwork/java/javase/downloads/jdk6downloads-1902814.html

- http://jmnote.com/wiki/VirtualBox_CentOS_6_%EC%84%A4%EC%B9%98

- http://wiki.centos.org/HowTos/Virtualization/VirtualBox

- VM

hiveproxy

hive1 (mysql master)

hive2 (mysql slave)

namenode

secondarynamenode

datanode1

datanode2

datanoe3

- VM 이미지 생성 후 각 이미지의 /etc/hosts 에 DNS 등록

- 브릿지, nat, 호스트전용 : http://searchme.tistory.com/16

- 브릿지, 호스트전용 : http://blog.naver.com/PostView.nhn?blogId=icysword&logNo=140135319512

- 첫번째와 같이 해도 되지만 개발환경에서 약간 통신 상에 문제가 있는 듯 하여 순서를 아래와 같이 변경 합니다.

- 1. NAT / 2. 브릿지 / 3. 호스트전용

- VM 이미지를 하나 만들어서 설치에 필요한 파일과 환경 구성을 한 다음 복제(완전복제)를 하십시오.

- hadoop user/group 생성

. https://www.google.co.kr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=8&cad=rja&ved=0CFoQtwIwBw&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DNKxjmDQYfJk&ei=tQh6UeyWJ-eOigeG4oCADw&usg=AFQjCNFJjajIFd7SrrgNuG5UdW0L7OECLQ&sig2=-28omSER_InldSy-yyIqaw&bvm=bv.45645796,d.aGc

- sudo groupadd hadoop

- sudo useradd hadoop -m -b /home/hadoop -d /home/hadoop -s /bin/bash

- sudo vi /etc/passwd 

        - 아래 설정의 경우 shadow password 에 대한 내용으로 계정 자체에 암호 생성을 하지 않을 경우 아래 설정은 진행 하지않아도 무방 함.

From

hadoop:*:1:1:hadoop Services:/home/hadoop:/bin/bash

To

hadoop::1:1:hadoop Services:/home/hadoop:/bin/bash

- ssh 연결을 위한 keygen (hadoop 계정)

http://jjeong.tistory.com/454

- hadoop 압축해제

. 설치경로 : /home/hadoop/app/hadoop

- JAVA_HOME 설정

vi conf/hadoop-env.sh

export JAVA_HOME=/home/hadoop/app/jdk

- core-site.xml 설정

<property>

<name>fs.default.name</name>

<value>hdfs://namenode:9000</value>

</property>

- hdfs-site.xml

<property>

<name>fs.default.name</name>

<value>hdfs://namenode:9000</value>

</property>

<property>

<name>dfs.name.dir</name>

<value>/home/hadoop/app/hadoop/data/name</value>

</property>

<property>

<name>dfs.data.dir</name>

<value>/home/hadoop/app/hadoop/data/hdfs</value>

</property>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.http.address</name>

<value>http://namenode:50070</value>

</property>

<property>

<name>dfs.secondary.http.address</name>

<value>http://secondarynamenode:50090</value>

</property>

- masters

. secondarynamenode 서버 정보를 작성한다.

secondarynamenode

- slaves

. datanode 서버 정보를 작성한다.

datanode1

datanode2

datanode3

- mapred-site.xml

<property>

<name>mapred.map.child.java.opts</name>

<value>-Xmx200m</value>

</property>

<property>

<name>mapred.reduce.child.java.opts</name>

<value>-Xmx200m</value>

</property>

<property>

<name>mapred.job.tracker</name>

<value>namenode:9001</value>

</property>

<property>

<name>mapred.system.dir</name>

<value>/home/hadoop/app/hadoop/mapred/system</value>

</property>

<property>

<name>mapred.local.dir</name>

<value>/home/hadoop/app/hadoop/mapred/tmp</value>

</property>

- namenode 에 구성한 모든 정보를 그대로 복사한다.

- VM 이미지 완전복제를 통해서 그대로 구성해도 됨.

- 스크립트 실행은 namenode 에서만 수행 한다.

- (최초 실행 시) bin/hadoop namenode -format

- bin/start-all.sh

- 모든 구성 서버들이 정상적으로 실행 되었는지 로그 확인.

- namenode, secondarynamenode, datanode, jobtracker, tasktracker 데몬이 떠야 함.

- http://namenode:50070/dfshealth.jsp

- http://namenode:50030/jobtracker.jsp

- bin/stop-all.sh

- 설치 경로와 mysql user 만 변경해서 설치 (아래 링크 참고)

- http://www.yongbok.net/blog/how-to-install-mysql-5-5-x-from-source/

- http://blog.beany.co.kr/archives/2648

- 설치 경로 : /home/hadoop/app/mysql

- mysql user : hadoop

cmake \

-DCMAKE_INSTALL_PREFIX=/home/hadoop/app/mysql \

-DMYSQL_UNIX_ADDR=/home/hadoop/app/mysql/mysql.sock \

-DDEFAULT_CHARSET=utf8 \

-DDEFAULT_COLLATION=utf8_general_ci \

-DWITH_EXTRA_CHARSETS=all \

-DMYSQL_DATADIR=/home/hadoop/app/mysql/db \

-DENABLED_LOCAL_INFILE=1 \

-DWITH_MYISAM_STORAGE_ENGINE=1 \

-DWITH_INNOBASE_STORAGE_ENGINE=1 \

-DWITH_ARCHIVE_STORAGE_ENGINE=1 \

-DWITH_BLACKHOLE_STORAGE_ENGINE=1 \

-DWITH_MEMORY_STORAGE_ENGINE=1 \

-DWITH_READLINE=1 \

-DENABLED_LOCAL_INFILE=1 \

-DMYSQL_USER=hadoop \

-DMYSQL_TCP_PORT=3306 \

-DENABLE_DOWNLOADS=1

make && make install

- 위치 : /home/hadoop/app/mysql/my.cnf

- vi my.cnf

basedir=/home/hadoop/app/mysql

datadir=/home/hadoop/app/mysql/db

port=3306

- sudo cp my.cnf /etc/my.cnf

- scripts/mysql_install_db --user=hadoop --defaults-file=//home/hadoop/app/mysql/my.cnf

- defaults-file 을 지정하지 않을 경우 /etc/my.cnf 파일 참조함.

- bin/mysqld_safe &

- bin/mysql -uroot

- use mysql;

- insert into user values(....);

- update user set password=password('....') where user='xxxx' and host='xxxx'

- flush privileges;

- http://blog.beany.co.kr/archives/1519

- hive 설치는 hadoop 이 설치된 서버와 별도로 설치 해도 됨.

- 단, hadoop 과 통신이 가능한 network 상에 있어야 함.

- 설치 경로 : /home/hadoop/app/hive

- namenode 에서 shell 로 실행

- bin/hadoop fs -mkdir /tmp

- bin/hadoop fs -mkdir /user/hive/warehouse

- bin/hadoop fs -chmod g+w /tmp

- bin/hadoop fs -chmod g+w /user/hive/warehouse

- cp conf/hive-env.sh.template conf/hive-env.sh

HADOOP_HOME=/home/hadoop/app/hadoop

- conf/hive-site.xml

<property>

<name>hive.metastore.warehouse.dir</name>

<value>/usr/hive/warehouse</value>

<description>hdfs path for hive metastore</description>

</property>

<property>

<name>fs.default.name</name>

<value>hdfs://namenode:9000</value>

</property>

<property>

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>

<description>JDBC connect string for a JDBC metastore</description>

</property>

<property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

<description>Driver class name for a JDBC metastore</description>

</property>

<property>

<name>javax.jdo.option.ConnectionUserName</name>

<value>hadoop</value>

<description>username to use against metastore database</description>

</property>

<property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>hadoop.user.mysql.password</value>

<description>password to use against metastore database</description>

</property> 

<property>

<name>datanucleus.autoCreateSchema</name>

<value>false</value>

</property> 

<property>

<name>datanucleus.fixedDatastore</name>

<value>true</value>

</property> 

- http://www.mysql.com/downloads/connector/j/

- mysql-connector-java-5.1.24.jar 를 hive/lib 아래로 복사

- bin/start-hive.sh

#!/bin/bash

HIVE_HOME=/home/hadoop/app/hive

/usr/bin/nohup $HIVE_HOME/bin/hive --service hiveserver > $HIVE_HOME/logs/hive.log 2>&1 < /dev/null &

echo $! > $HIVE_HOME/bin/hive-server.pid

- bin/stop-hive.sh

#!/bin/bash

/bin/cat /home/hadoop/app/hive/bin/hive-server.pid | xargs kill -9

- bin/hive

hive> quit;

- mysql -uroot

mysql> create database metastore_db;

mysql> use metastore_db;

mysql> source /home/hadoop/app/hive/scripts/metastore/upgrade/mysql/hive-schema-0.9.0.mysql.sql

mysql> show tables;

- vi textfile

1,2

3,4

5,6

- bin/hive

hive> create table citi (citing int, cited int) row format delimited fields terminated by ',' stored as textfile;

- mysql -uroot

mysql> use metastore_db;

mysql> select * from TBLS\G;

- hive 1번 서버 구성과 같은 구성으로 hive 2번 서버를 구성한다.

- 구성된 서버들을 haproxy 로 묶는다.

- http://haproxy.1wt.eu/

- https://github.com/langpavel/haproxy-doc

- http://linuxadminzone.com/install-and-configure-haproxy-the-software-based-loadbalancer-in-ubuntu/

- http://www.sestory.net/73

- http://helloworld.naver.com/helloworld/284659

global

        log 127.0.0.1  local0

        maxconn 1024

        uid 99

        gid 99

        daemon

defaults

        log    global

        mode    http

        option httplog

        option dontlognull

        retries 3

        option redispatch

        maxconn 1024

        contimeout      5000

        clitimeout      50000

        srvtimeout      50000

listen  hive-service 192.168.56.102:80

        option forwardfor

        option httpchk OPTIONS /health_check.html

        stats enable

        stats refresh 10s 

        stats hide-version

        stats scope   .   

        server  web1 www.naver.com:80 check inter 2000 fall 3

        server  web2 www.daum.net:80 check inter 2000 fall 3

- HAProxy : hive service HA 구성을 위해서 사용

- Hive : hadoop 에 데이터 적재 및 분석

- MySQL : hive 의 meta 정보를 저장

- Hadoop : 분산파일시스템 및 MR 관리

......

elif [ "$COMMAND" = "fs" ] ; then

  CLASS=org.apache.hadoop.fs.FsShell

  HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"

elif [ "$COMMAND" = "dfs" ] ; then

  CLASS=org.apache.hadoop.fs.FsShell

  HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"

......

Unable to load realm info from SCDynamicStore

this line : # export HADOOP_OPTS=-server

to : export HADOOP_OPTS="-Djava.security.krb5.realm= -Djava.security.krb5.kdc="

Using Remote Login to Start the SSH Server in Mac OS X

These instructions are the same for OS X 10.7 Lion and 10.6:

• Open System Preferences and click on “Sharing”
• Select the checkbox next to “Remote Login” to enable it, like the screenshot above

Error: JAVA_HOME is not set.

line : # export JAVA_HOME=/usr/lib/j2sdk1.5-sun

to : export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/HOME