Hadoop 클러스터 설치 (고가용성) - 1

HDFS 고가용성 구성

QJM(Quorum Journal Manager)

이미지 출처: https://wikidocs.net/23628

 

주키퍼 설치

1. 다운로드 및 압축해제

hadoop@hadoop-001:~$ sudo curl -o /usr/local/apache-zookeeper-3.7.0-bin.tar.gz http://apache.tt.co.kr/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.8M  100 11.8M    0     0  4001k      0  0:00:03  0:00:03 --:--:-- 4003k
hadoop@hadoop-001:~$ sudo mkdir -p /usr/local/zookeeper && sudo tar -xvzf /usr/local/apache-zookeeper-3.7.0-bin.tar.gz -C /usr/local/zookeeper --strip-components 1
apache-zookeeper-3.7.0-bin/docs/
apache-zookeeper-3.7.0-bin/docs/skin/
...

2. 샘플 설정 파일 복사

hadoop@hadoop-001:~$ cp -av /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg
'/usr/local/zookeeper/conf/zoo_sample.cfg' -> '/usr/local/zookeeper/conf/zoo.cfg'

3. 환경 파일 설정

주키퍼 환경 파일을 설정한다.

hadoop@hadoop-001:~$ vi /home/apps/zookeeper/conf/zoo.cfg

...
dataDir=/usr/local/zookeeper/data

# server
server.1=hadoop-001:2888:3888
server.2=hadoop-002:2888:3888
server.3=hadoop-003:2888:3888
...

JVM 메모리를 설정한다.

hadoop@hadoop-001:~$ vi /usr/local/zookeeper/conf/java.env
export JVMFLAGS="-Xmx2G"

4. 데이터 및 로그 폴더 생성

hadoop@hadoop-001:~$ mkdir -p /usr/local/zookeeper/data /usr/local/zookeeper/run usr/local/zookeeper/logs

5. 초기 설정 스크립트 작성

MYID 변수에는 서버마다 유니크한 아이디 지정한다.

hadoop@hadoop-001:~$ sudo vi /usr/bin/zookeeper-initialize
#!/bin/bash

MYID=1 # 서버마다 유니크한 아이디 지정해야함
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export ZOOPIDFILE=${ZOOPIDFILE:-/usr/local/zookeeper/run/zookeeper-server.pid}
export ZOOKEEPER_HOME=${ZOOKEEPER_HOME:-/usr/local/zookeeper}
export ZOOKEEPER_CONF=${ZOOKEEPER_CONF:-/usr/local/zookeeper/conf}
export ZOOCFGDIR=${ZOOCFGDIR:-$ZOOKEEPER_CONF}
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_CONF:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
export ZOO_LOG_DIR=${ZOO_LOG_DIR:-/usr/local/zookeeper/logs}
export ZOO_LOG4J_PROP=${ZOO_LOG4J_PROP:-INFO,ROLLINGFILE}
export JVMFLAGS=${JVMFLAGS:--Dzookeeper.log.threshold=INFO}
export ZOO_DATADIR_AUTOCREATE_DISABLE=${ZOO_DATADIR_AUTOCREATE_DISABLE:-true}
env CLASSPATH=$CLASSPATH $ZOOKEEPER_HOME/bin/zkServer-initialize.sh --myid=$MYID

6. 실행 스크립트 작성

hadoop@hadoop-001:~$ sudo vi /usr/bin/zookeeper-server
#!/bin/bash

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export ZOOPIDFILE=${ZOOPIDFILE:-/usr/local/zookeeper/run/zookeeper-server.pid}
export ZOOKEEPER_HOME=${ZOOKEEPER_HOME:-/usr/local/zookeeper}
export ZOOKEEPER_CONF=${ZOOKEEPER_CONF:-/usr/local/zookeeper/conf}
export ZOOCFGDIR=${ZOOCFGDIR:-$ZOOKEEPER_CONF}
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_CONF:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
export ZOO_LOG_DIR=${ZOO_LOG_DIR:-/usr/local/zookeeper/logs}
export ZOO_LOG4J_PROP=${ZOO_LOG4J_PROP:-INFO,ROLLINGFILE}
export JVMFLAGS=${JVMFLAGS:--Dzookeeper.log.threshold=INFO}
export ZOO_DATADIR_AUTOCREATE_DISABLE=${ZOO_DATADIR_AUTOCREATE_DISABLE:-true}
env CLASSPATH=$CLASSPATH $ZOOKEEPER_HOME/bin/zkServer.sh "$@"

 

7. 기동 쉘 스크립트 작성

hadoop@hadoop-001:~$ sudo vi /etc/init.d/zookeeper
#!/bin/bash

### BEGIN INIT INFO
# Provides:          zookeeper
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: zookeeper service
### END INIT INFO
  
ZK_HOME=${ZOOKEEPER_HOME:-/usr/local/zookeeper}
ZK_INIT_SCRIPT="/usr/bin/zookeeper-initialize"
ZK_DAEMON_SCRIPT="/usr/bin/zookeeper-server"
ZK_PID_FILE=$ZK_HOME/run/zookeeper-server.pid

DESC="zookeeper daemon"

case "$1" in
    init)
        if [ -f "$ZK_PID_FILE" ] ; then
            echo "Error: $DESC is running. Stop it first."
            exit 1
        else
            ${ZK_INIT_SCRIPT} $*
        fi
        ;;
    start)
        ${ZK_DAEMON_SCRIPT} start
        ;;
    stop)
        if [ -f "$ZK_PID_FILE" ] ; then
            ${ZK_DAEMON_SCRIPT} stop
        else
            echo "Already $DESC is running."
        exit 1        
        fi
        ;;
    forcestop)
        echo -n "force stopping $DESC: "
        if [ -f "$ZK_PID_FILE" ] ; then
            PID=`cat "$ZK_PID_FILE" 2>/dev/null`
            kill -9 "$PID"
        else
            echo "ERROR."
        fi
        ;;
    restart)
        $0 stop
        sleep 2
        $0 start
        ;;
    status)
        if [ -f "$ZK_PID_FILE" ] ; then
            echo "running"
        else
            echo "not running."
            exit 1
        fi
        ;;
    *)        
        echo "Usage: start|stop|restart|status|forcestop}" >&2  
        exit 1
        ;;
esac
exit 0

8. 작성한 스크립트 권한 부여

hadoop@hadoop-001:~$ sudo chmod 755 /usr/bin/zookeeper-server /usr/bin/zookeeper-initialize /etc/init.d/zookeeper

9. 서비스 등록

서비스를 등록하여 부팅시 자동으로 실행되게 설정하여준다.

root@hadoop-001:~# cat > /etc/systemd/system/zookeeper.service
[Unit]
Description=Zookeeper Server

[Service]
Type=simple
User=hadoop
ExecStart=/etc/init.d/zookeeper start
ExecStop=/etc/init.d/zookeeper stop
ExecReload=/etc/init.d/zookeeper restart
PIDFile=/usr/local/zookeeper/run/zookeeper-server.pid

[Install]
WantedBy=multi-user.target

10. 서비스 데몬 재시작

root@hadoop-001:~# systemctl daemon-reload
root@hadoop-001:~# systemctl enable zookeeper
Created symlink /etc/systemd/system/multi-user.target.wants/zookeeper.service → /etc/systemd/system/zookeeper.service.

 

주키퍼 실행/중지

1. 초기 설정 실행

hadoop@hadoop-001:~$ /etc/init.d/zookeeper-server init
Using myid of 1

2. 실행

root@hadoop-001:~# systemctl start zookeeper

3. 중지

root@hadoop-001:~# systemctl stop zookeeper

4. 재시작

root@hadoop-001:~# systemctl restart zookeeper

5. 상태

hadoop@hadoop-003:~$ systemctl status zookeeper.service 
● zookeeper.service - Zookeeper Server
     Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-12-19 19:29:27 KST; 14h ago
   Main PID: 1745 (java)
      Tasks: 37 (limit: 4556)
     Memory: 63.5M
     CGroup: /system.slice/zookeeper.service
             └─1745 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.datadir.autocreate=false -Dzookeeper.log.dir=/usr/local/zookeeper/logs -Dzookeeper.log.file=zookeeper-hadoop-server-hadoop-003.log -Dzookeeper.root.log>

12월 19 19:29:27 hadoop-003 systemd[1]: Started Zookeeper Server.
12월 19 19:29:27 hadoop-003 zookeeper[1728]: ZooKeeper JMX enabled by default
12월 19 19:29:27 hadoop-003 zookeeper[1728]: Using config: /usr/local/zookeeper/conf/zoo.cfg
12월 19 19:29:28 hadoop-003 zookeeper[1728]: Starting zookeeper ... STARTED
lines 1-13/13 (END)