'snappy' 태그의 글 목록

'snappy'에 해당되는 글 2건

[Hadoop] Native library installation on osx

Elastic/Hadoop 2017. 11. 21. 15:54

hdfs 에서 snappy 압축 파일을 바로 읽으려고 하니 아래와 같은 오류 메시지가 발생을 했습니다.

hadoop 을 구성 할 때 source build 를 하지 않고 그냥 binary 를 가지고 사용해서 그런것 같아 source 를 받아서 build 를 하기로 했습니다.

[에러 메시지]

Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

$ hadoop checknative -a

Native library checking:

hadoop: false

zlib: false

snappy: false

lz4: false

bzip2: false

openssl: false

처럼 나와서 빌드를 하게 되었습니다.

참고문서는 아래 링크를 보시면 됩니다.

Ref.

https://medium.com/@faizanahemad/hadoop-native-libraries-installation-on-mac-osx-d8338a6923db

https://gist.github.com/zedar/f631ace0759c1d512573

brea install 을 통해서 필요한 몇 가지를 먼저 구성 하셔야 합니다.

$ brew install gcc autoconf automake libtool cmake snappy gzip bzip2 homebrew/versions/protobuf250 zlib openssl

참고로 저는 protobuf 3.4.0 이 설치되어 있어서 downgrade 했습니다.

pure hadoop build 는 maven 은 3.x 이상을 요구 합니다.

- hadoop-2.7.2 빌드 했습니다.

[빌드 및 native library 복사]

$ ../apache-maven-3.5.0/bin/mvn package -Pdist,native -DskipTests -Dtar -e

$ vi .bash_profile

export OPENSSL_ROOT_DIR=/usr/local/Cellar/openssl/1.0.2m

export OPENSSL_INCLUDE_DIR=/usr/local/Cellar/openssl/1.0.2m/include

export PROTOC_HOME=/usr/local/opt/protobuf@2.5

export HADOOP_HOME=/Users/henry/Work/apps/hadoop-2.7.2

PATH=$PROTOC_HOME/bin:$HADOOP_HOME/bin:$HOME/bin:$PATH

export PATH

$ cp -r hadoop-dist/target/hadoop-2.7.2/lib/native/* /Users/henry/Work/apps/hadoop-2.7.2/lib/native/osx/

$ vi etc/hadoop/hadoop-env.sh

export HADOOP_HOME="/Users/henry/Work/apps/hadoop-2.7.2"

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native/osx"

$ vi etc/hadoop/core-site.xml

<name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.BZip2Codec</value>

</property>

여기 까지 하고 나서 아래 명령어를 다시 실행해 봅니다.

$ hadoop checknative -a

17/11/21 15:45:16 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version

17/11/21 15:45:16 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library

Native library checking:

hadoop: true /Users/henry/Work/apps/hadoop-2.7.2/lib/native/osx/libhadoop.dylib

zlib: true /usr/lib/libz.1.dylib

snappy: true /usr/local/lib/libsnappy.1.dylib

lz4: true revision:99

bzip2: false

openssl: false build does not support openssl.

Hadoop source build 하다 보면 아래 에러가 발생을 합니다.

[INFO] Apache Hadoop Pipes ................................ FAILURE [  0.627 s]

이 경우 아래 문서 참고해서 해결 하시면 됩니다.

Ref.

https://stackoverflow.com/questions/36818957/mac-hadoop-2-7-failed-to-execute-goal-org-apache-maven-pluginsmaven-antrun

$ cd hadoop/hadoop-tools/hadoop-pipes

$ vi pom.xml

...중략...

...중략...

-> 여기서 64를 6으로 변경해 주시기 바랍니다.

저작자표시 비영리 변경금지

[HADOOP] snappy 압축 파일 읽기.

ITWeb/Hadoop일반 2014. 7. 15. 12:55

테스트를 하다 보면 hdfs 에 snappy 로 압축 된 파일을 읽어서 내용을 검증 하는 기능을 넣을때가 있습니다.

ㅎㅎ 제가 구현 하던 모듈에 필요해서 공유 합니다.

기본 구조는 아래 링크 참고 해서 작성 하였습니다.

[참조소스]

https://code.google.com/p/hadoop-snappy/source/browse/trunk/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCodec.java

* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements.  See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership.  The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License.  You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.hadoop.io.compress.snappy;

import java.io.BufferedReader;
import java.io.File;
import java.io.InputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;

import junit.framework.TestCase;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.compress.CompressionCodec;
import org.apache.hadoop.io.compress.CompressionInputStream;
import org.apache.hadoop.io.compress.CompressionOutputStream;
import org.apache.hadoop.io.compress.SnappyCodec;
import org.apache.hadoop.util.ReflectionUtils;

public class TestSnappyCodec extends TestCase {
  private String inputDir;

  @Override
  protected void setUp() throws Exception {
  super.setUp();
  inputDir = System.getProperty("test.build.data", "target");
  }

  public void testFile() throws Exception {
  run("test.txt");
  }

  private void run(String filename) throws FileNotFoundException, IOException{
  File snappyFile = new File(inputDir, filename + new SnappyCodec().getDefaultExtension());
  if (snappyFile.exists()) {
  snappyFile.delete();
  }

  Configuration conf = new Configuration();
  CompressionCodec codec = (CompressionCodec) ReflectionUtils.newInstance(SnappyCodec.class, conf);

  // Compress
  InputStream is = this.getClass().getClassLoader().getResourceAsStream("test.txt");
  FileOutputStream os = new FileOutputStream(snappyFile);
  CompressionOutputStream cos = codec.createOutputStream(os);

  byte buffer[] = new byte[8192];
  try {
  int bytesRead = 0;
  while ((bytesRead = is.read(buffer)) > 0) {
  cos.write(buffer, 0, bytesRead);
  }
  } catch (IOException e) {
  System.err.println("Compress Error");
  e.printStackTrace();
  } finally {
  is.close();
  cos.close();
  os.close();
  }

  // Decompress
  is =  this.getClass().getClassLoader().getResourceAsStream("test.txt");
  FileInputStream is2 = new FileInputStream(snappyFile);
  CompressionInputStream cis = codec.createInputStream(is2);
  BufferedReader r = new BufferedReader(new InputStreamReader(is));
  BufferedReader cr = new BufferedReader(new InputStreamReader(cis));


  try {
  String line, rline;
  int lineNum = 0;
  while ((line = r.readLine()) != null) {
  lineNum++;
  rline = cr.readLine();
  if (!rline.equals(line)) {
  System.err.println("Decompress error at line " + line + " of file " + filename);
  System.err.println("Original: [" + line + "]");
  System.err.println("Decompressed: [" + rline + "]");
  }
  assertEquals(rline, line);
  }
  assertNull(cr.readLine());
  } catch (IOException e) {
  System.err.println("Decompress Error");
  e.printStackTrace();
  } finally {
  cis.close();
  is.close();
  os.close();
  }
  }
}

[구현소스]

- 위 예제는 로컬테스트용으로 작성 되어 있기 때문에 hdfs 에서 직접 읽어오는 형태로 약간 수정했습니다.

public static void readLines(Path location, Configuration conf) throws Exception {

    CompressionCodec codec = (CompressionCodec) ReflectionUtils.newInstance(SnappyCodec.class, conf);
    FileSystem hdfs = FileSystem.get(conf);

    try{
      FileStatus[] fileStatus = hdfs.listStatus(location);

      for (FileStatus file : fileStatus) {
        if ( file.isDirectory() ) {
          continue;
        }

        CompressionInputStream cis = codec.createInputStream(hdfs.open(file.getPath()));
        BufferedReader br = new BufferedReader(new InputStreamReader(cis));

        try {
          String line;
          line=br.readLine();

          while (line != null){
            line = br.readLine();

System.out.println(line);
          }
        } finally {
          br.close();
        }
      }
    } catch ( IOException e ) {
    }
  }

◀ PREV : [1] : NEXT ▶

jjeong

'snappy'에 해당되는 글 2건

[Hadoop] Native library installation on osx

[HADOOP] snappy 압축 파일 읽기.

티스토리툴바