kafka

카프카 설치

1. 다운로드 및 압축해제

hadoop@hadoop-001:~$ sudo curl -o /usr/local/kafka_2.13-3.0.0.tgz http://mirror.apache-kr.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 82.3M  100 82.3M    0     0  2716k      0  0:00:31  0:00:31 --:--:-- 2397k
hadoop@hadoop-001:~$ sudo mkdir -p /usr/local/kafka && sudo tar -xvzf /usr/local/kafka_2.13-3.0.0.tgz -C /usr/local/kafka --strip-components 1
kafka_2.13-3.0.0/LICENSE
kafka_2.13-3.0.0/NOTICE
kafka_2.13-3.0.0/bin/
...

2. 환경 파일 설정

log 디렉토리 설정, 연결 대상 주키퍼 호스트 변경 등을 설정한다.

hadoop@hadoop-001:~$ vi /usr/local/kafka/config/server.properties

...
log.dirs=/usr/local/kafka/logs
num.partitions=3

# root directory for all kafka znodes.
zookeeper.connect=hadoop-001:2181,hadoop-002:2181,hadoop-003:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=60000

delete.topic.enable=true
default.replication.factor=1
...

 

hadoop@hadoop-001:~$ vi /usr/local/kafka/config/producer.properties
...
bootstrap.servers=hadoop-001:9092,hadoop-002:9092,hadoop-003:9092
...

3. 로그 폴더 생성

hadoop@hadoop-001:~$ mkdir /usr/local/kafka/logs

 

4. 기동 쉘 스크립트 작성

hadoop@hadoop-001:~$ sudo vi /etc/init.d/kafka
#!/bin/bash

### BEGIN INIT INFO
# Provides:          kafka
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: kafka service
### END INIT INFO

KAFKA_HOME=/usr/local/kafka
KAFKA_PID_FILE=$KAFKA_HOME/kafka.pid
 
if [ "$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx5G -Xms5G"
fi
  
case "$1" in
 
  start)
        echo -n "Starting Kafka: "
        if [ -f $KAFKA_PID_FILE ] && [ "PID" = "" ] ; then
            rm -rf $KAFKA_PID_FILE
            echo "Deleted $KAFKA_PID_FILE"
        fi	
        if [ -f $KAFKA_PID_FILE ] ; then
            echo "Already started"
            exit 3		
        else
            nohup $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties > $KAFKA_HOME/logs/server.out 2> $KAFKA_HOME/logs/server.err &			
            sleep 2            
            PID=`ps -ef | grep kafka.Kafka | grep -v grep | awk '{print $2}'`
            echo $PID > $KAFKA_PID_FILE
            echo "done."
            exit 0
        fi
        ;;
 
  stop)
        echo -n "Stopping Kafka: "
        if [ -f $KAFKA_PID_FILE ] ; then
            PID=`cat "$KAFKA_PID_FILE" 2>/dev/null`
            kill -9 $PID
	    rm -rf $KAFKA_PID_FILE
            sleep 2
            echo "done."
            exit 0		
        else
            echo "Already stopped"
            exit 3
        fi
        ;;
 
  status)
        if [ -f $KAFKA_PID_FILE ] ; then
            PID=`cat "$KAFKA_PID_FILE" 2>/dev/null`
            echo "Running $PID"
            exit 0
        else
            echo "Stopped"
            exit 3		
        fi
        ;;
 
  restart)
        $0 stop
        sleep 2
        $0 start
        ;;
 
  *)
        echo "Usage: $0 {start|stop|status|restart}"
        exit 1
        ;;
 
esac

5. 작성한 스크립트 권한 부여

hadoop@hadoop-001:~$ sudo chmod 755 /etc/init.d/kafka

6. 서비스 등록

서비스를 등록하여 부팅시 자동으로 실행되게 설정하여준다.

주키퍼 다음에 실행하도록 설정한다.

root@hadoop-001:~# cat > /etc/systemd/system/kafka.service
[Unit]
Description=Kafka Server
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=hadoop
ExecStart=/etc/init.d/kafka start
ExecStop=/etc/init.d/kafka stop
ExecReload=/etc/init.d/kafka restart
PIDFile=/usr/local/kafka/kafka.pid

[Install]
WantedBy=multi-user.target

7. 서비스 데몬 재시작

root@hadoop-001:~# systemctl daemon-reload
root@hadoop-001:~# systemctl enable kafka
Synchronizing state of kafka.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable kafka
Created symlink /etc/systemd/system/multi-user.target.wants/kafka.service → /etc/systemd/system/kafka.service.

 

카프카 실행/중지

1. 실행

root@hadoop-001:~# systemctl start kafka

2. 중지

root@hadoop-001:~# systemctl stop kafka

3. 재시작

root@hadoop-001:~# systemctl restart kafka

4. 상태

adoop@hadoop-001:~$ systemctl status kafka.service 
● kafka.service - Kafka Server
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-12-19 19:22:30 KST; 14h ago
   Main PID: 1405 (java)
      Tasks: 69 (limit: 4556)
     Memory: 364.5M
     CGroup: /system.slice/kafka.service
             └─1405 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xloggc:/usr/local/kafka>

12월 19 19:22:30 hadoop-001 systemd[1]: Started Kafka Server.
12월 19 19:22:30 hadoop-001 kafka[1404]: Starting Kafka: done.