Kafka์—์„œ ์ œ๊ณตํ•˜๋Š” 42๊ฐœ์˜ Shell Script ๋„๊ตฌ ์ „๋ถ€ ์‚ดํŽด๋ณด๊ธฐ ๐Ÿ”

8 minute read

๋“ค์–ด๊ฐ€๋ฉฐ

ํšŒ์‚ฌ์—์„œ Confluent๋ฅผ ํ†ตํ•ด Kafka ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ์ž˜ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฐ๋ฐ, ์ด ์นดํ”„์นด์— ๋Œ€ํ•ด์„œ ์ข€๋” ์ž์„ธํžˆ ์•Œ๊ณ , ์ „๋ฌธ์„ฑ์„ ๊ฐ–์ถ”๊ณ  ์‹ถ๋‹ค๋Š” ์ƒ๊ฐ์ด ๋“ค์–ด์„œ 2025๋…„ ์ฒซ ๋ชฉํ‘œ๋กœ Confluent์˜ Kafka ์ž๊ฒฉ์ฆ์ธ Confluent Certified Developer for Apache Kafkaยฎ ์ž๊ฒฉ์ฆ์„ ์ค€๋น„ํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค โœŒ๏ธ

์ด๋ฒˆ ํฌ์ŠคํŠธ๋Š” ์ž‘์„ฑ ์‹œ์  ๊ธฐ์ค€ ์ตœ์‹  ๋ฒ„์ „์ธ Kafka 3.9์—์„œ ์ œ๊ณตํ•˜๋Š” ๋ชจ๋“  Kafka Shell Script์˜ ๊ธฐ๋Šฅ์„ ํ•œ๋ฒˆ์”ฉ ์‚ดํŽด๋ณด๋Š” ๊ฑธ ๋ชฉํ‘œ๋กœ ๊ธฐ๋กํ•œ ํฌ์ŠคํŠธ ์ž…๋‹ˆ๋‹ค.

With Authentication

๋Œ€๋ถ€๋ถ„ ๋กœ์ปฌ K8s์— ๋””ํ”Œ๋กœ์ด ํ•œ Kafka ํด๋Ÿฌ์Šคํ„ฐ์— ์‹คํ–‰ํ•ด๋ณด์•˜๊ณ , ์ผ๋ถ€ ๊ฒฝ์šฐ๋Š” Confluent Cluster์—์„œ ์‹คํ—˜ํ•ด๋ณด์•˜์Šต๋‹ˆ๋‹ค.

๋กœ์ปฌ์— ๋””ํ”Œ๋กœ์ด ํ•œ ์นดํ”„์นด ํด๋Ÿฌ์Šคํ„ฐ์—์„œ๋Š” ํŽธํ•˜๊ฒŒ ์‹คํ—˜ํ•ด๋ณด๊ธฐ ์œ„ํ•ด์„œ Authentication ๊ณผ์ •์„ ์—†์•ด์Šต๋‹ˆ๋‹ค! ๋งŒ์•ฝ, SASL ์ธ์ฆ์„ ํ•ด์•ผ ํ•œ๋‹ค๋ฉด, Shell ์‹คํ–‰ ๋•Œ ์•„๋ž˜์™€ ๊ฐ™์ด ์ถ”๊ฐ€๋กœ ์„ค์ •ํ•ด์ค์‹œ๋‹ค.

$ kafka-xxxx.sh \
  --bootstrap-servers xxxx:9092 \
  --command-config /tmp/xxxx.properties

Kafka Shell์€ ๋กœ์ปฌ์—์„œ ์‹คํ–‰ํ•  ์ˆ˜๋„ ์žˆ์„ ๊ฒƒ์ด๊ณ , Kafka Shell์„ ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ๋Š” ์ปจํ…Œ์ด๋„ˆ์— kubectl exec๋กœ ์ ‘์†ํ•ด ์‹คํ–‰ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ œ๊ฐ€ CCDAK ์ž๊ฒฉ์ฆ์„ ์ค€๋น„ํ•˜๋ฉด์„œ ๋Š๋‚€ ์ ์€ โ€œ์ปจํ…Œ์ด๋„ˆ๋กœ ๊ฒฉ๋ฆฌ๋œ ํ™˜๊ฒฝ์—์„œ Kafka Shell์„ ์‹คํ–‰ํ•˜๋Š”๊ฒŒ ๊ฐ€์žฅ ์•ˆ์ „ํ•˜๋‹คโ€œ๋Š” ๊ฒƒ ์ž…๋‹ˆ๋‹ค. ๋งฅ๋ถ ๋กœ์ปฌ์˜ Kafka Shell์—์„œ๋Š” Exception์„ ๋ฟœ์œผ๋ฉฐ ์ œ๋Œ€๋กœ ๋™์ž‘ํ•˜์ง€ ์•Š๋˜ ๊ฒƒ์ด Kafka Broker pod์—์„œ๋Š” ์ œ๋Œ€๋กœ ๋™์ž‘ํ•˜๋Š” ๊ฒฝ์šฐ๊ฐ€ ์ข…์ข… ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค!

์ž…๋ฌธ

  • kafka-topics.sh
  • kafka-configs.sh
    • ์ด๋ฏธ ์ƒ์„ฑํ•œ ์นดํ”„์นด ํ† ํ”ฝ์˜ Config๋ฅผ ๋ณ€๊ฒฝํ•  ๋•Œ ์‚ฌ์šฉํ•จ. --alter๋ฅผ ์‚ฌ์šฉ.
    • โ€œKafka Shell: kafka-topics.shโ€ ํฌ์ŠคํŠธ์— ํ•จ๊ป˜ ๊ธฐ์ˆ ํ•จ.
  • kafka-console-producer.sh
  • kafka-console-consumer.sh
  • kafka-consumer-groups.sh
    • --list
      • ์ปจ์Šˆ๋จธ ๊ทธ๋ฃน ๋ฆฌ์ŠคํŒ…
    • --describe --group xxxx
      • ์ปจ์Šˆ๋จธ ๊ทธ๋ฃน ์ƒ์„ธ ๋ณด๊ธฐ
    • --execute --to-latest/--to-earlist/...
      • ์ปจ์Šˆ๋จธ ๊ทธ๋ฃน์˜ offset์„ ์ดˆ๊ธฐํ™” ํ•˜๊ฑฐ๋‚˜ ์ตœ์‹ ์œผ๋กœ ์ด๋™ ์‹œํ‚ค๋ ค๊ณ  ํ•  ๋•Œ

Performance Test

์นดํ”„์นด ํด๋Ÿฌ์Šคํ„ฐ์˜ ํผํฌ๋จผ์Šค๋ฅผ ์ธก์ •ํ•˜๊ธฐ ์œ„ํ•œ ๋„๊ตฌ ์ž…๋‹ˆ๋‹ค.

$ kafka-producer-perf-test.sh \
    --producer-props bootstrap.servers=bitnami-kafka:9092 \
    --topic test \
    --num-records 100 \
    --throughput 10 \
    --record-size 100
52 records sent, 10.3 records/sec (0.00 MB/sec), 8.3 ms avg latency, 316.0 ms max latency.
100 records sent, 10.027073 records/sec (0.00 MB/sec), 4.84 ms avg latency, 316.00 ms max latency, 1 ms 50th, 13 ms 95th, 316 ms 99th, 316 ms 99.9th.
$ kafka-consumer-perf-test.sh \
    --bootstrap-server bitnami-kafka:9092 \
    --topic test \
    --messages 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2025-01-15 09:26:16:817, 2025-01-15 09:26:20:246, 0.0096, 0.0028, 105, 30.6212, 3411, 18, 0.5312, 5833.3333

Reassign Partitions

ํ† ํ”ฝ์„ ์ด๋ฃจ๋Š” ๋ฆฌ๋” ํŒŒํ‹ฐ์…˜์ด ํ•˜๋‚˜์˜ ๋ธŒ๋กœ์ปค์— ์˜ฌ๋ฆฌ๋Š” ํ•ซ์ŠคํŒŸ(Hot spot) ํ˜„์ƒ์„ ๋ฐฉ์ง€ํ•˜๊ธฐ ์œ„ํ•œ ๋„๊ตฌ ์ž…๋‹ˆ๋‹ค. kafka-reassign-partitions.sh ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด, ๋ฆฌ๋” ํŒŒํ‹ฐ์…˜๊ณผ ํŒ”๋กœ์›Œ ํŒŒํ‹ฐ์…˜ ์œ„์น˜๋ฅผ ์žฌ์„ค์ • ํ•ฉ๋‹ˆ๋‹ค.

๋จผ์ € ์žฌ๋ถ„๋ฐฐ ์ž‘์—…์„ ์œ„ํ•ด ์ ๋‹นํ•œ ํŒŒํ‹ฐ์…˜๊ณผ Replication Factor๋ฅผ ๊ฐ€์ง„ ํ† ํ”ฝ์„ ํ•˜๋‚˜ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค.

$ kafka-topics.sh \
  --bootstrap-server bitnami-kafka:9092 \
  --topic test2 \
  --partitions 3 \
  --replication-factor 3 \
  --create
$ kafka-topics.sh \
  --bootstrap-server bitnami-kafka:9092 \
  --topic test2 \
  --describe
Topic: test2    TopicId: a3vf0OqaSN6sqjtAVfrOhw PartitionCount: 3       ReplicationFactor: 3    Configs:
        Topic: test2    Partition: 0    Leader: 101     Replicas: 101,100,102   Isr: 101,100,102        Elr: N/A        LastKnownElr: N/A
        Topic: test2    Partition: 1    Leader: 100     Replicas: 100,102,101   Isr: 100,102,101        Elr: N/A        LastKnownElr: N/A
        Topic: test2    Partition: 2    Leader: 102     Replicas: 102,101,100   Isr: 102,101,100        Elr: N/A        LastKnownElr: N/A

์ด๊ฒŒ ํ˜„์žฌ ๋ฆฌ๋” ํŒŒํ‹ฐ์…˜์ด ๊ฐ ๋ธŒ๋กœ์ปค์— ๊ณ ๋ฅด๊ฒŒ ๋ถ„์‚ฐ ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋ฅผ 102๋ฒˆ ๋ธŒ๋กœ์ปค์— ์ง‘์ค‘์‹œ์ผœ ๊ฐ•์ œ๋กœ ํ•ซ์ŠคํŒŸ์œผ๋กœ ๋งŒ๋“ค์–ด๋ด…์‹œ๋‹ค! ์ผ๋‹จ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•˜๊ธฐ ์œ„ํ•ด์„  --reassignment-json-file์— ์ „๋‹ฌํ•  json ํŒŒ์ผ์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.

$ cat <<EOF > /tmp/partitions.json
{
  "partitions":
  [
    { "topic": "test2", "partition": 0, "replicas": [102,100,101] }
  ],
  "version": 1
}
EOF
$ kafka-reassign-partitions.sh \
  --bootstrap-server bitnami-kafka:9092 \
  --reassignment-json-file /tmp/partitions.json \
  --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"test2","partition":0,"replicas":[100,102,101],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignment for test2-0
$ kafka-reassign-partitions.sh \
  --bootstrap-server bitnami-kafka:9092 \
  --reassignment-json-file /tmp/partitions.json \
  --verify
Status of partition reassignment:
Reassignment of partition test2-0 is completed.

Clearing broker-level throttles on brokers 100,101,102
Clearing topic-level throttles on topic test2

๊ทธ๋ฆฌ๊ณ  ๋‹ค์‹œ --describe๋ฅผ ํ•ด๋ณด๋ฉด

$ kafka-topics.sh \
  --bootstrap-server bitnami-kafka:9092 \
  --topic test2 \
  --describe
Topic: test2    TopicId: a3vf0OqaSN6sqjtAVfrOhw PartitionCount: 3       ReplicationFactor: 3    Configs:
        Topic: test2    Partition: 0    Leader: 101     Replicas: 101,102,100   Isr: 101,100,102        Elr: N/A        LastKnownElr: N/A
        Topic: test2    Partition: 1    Leader: 100     Replicas: 100,101,102   Isr: 100,102,101        Elr: N/A        LastKnownElr: N/A
        Topic: test2    Partition: 2    Leader: 102     Replicas: 102,101,100   Isr: 102,101,100        Elr: N/A        LastKnownElr: N/A

์•„์ง ๋ฐ˜์˜์ด ์•ˆ ๋˜์—ˆ๋‹ค. ๋ฆฌ๋” ์žฌ์„ ์ถœ์„ ์œ„ํ•ด์„œ๋Š” kafka-leader-election.sh ๋ช…๋ น์–ด๋ฅผ ์ถ”๊ฐ€๋กœ ์‹คํ–‰ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

$ kafka-leader-election.sh \
  --bootstrap-server bitnami-kafka:9092 \
  --topic test2 \
  --partition 0 \
  --election-type preferred

๊ทธ๋ฆฌ๊ณ  ๋‹ค์‹œ --describe๋ฅผ ํ•ด๋ณด๋ฉด

$ kafka-topics.sh \
  --bootstrap-server bitnami-kafka:9092 \
  --topic test2 \
  --describe
Topic: test2    TopicId: a3vf0OqaSN6sqjtAVfrOhw PartitionCount: 3       ReplicationFactor: 3    Configs:
        Topic: test2    Partition: 0    Leader: 102     Replicas: 102,100,101   Isr: 101,100,102        Elr: N/A        LastKnownElr: N/A
        Topic: test2    Partition: 1    Leader: 100     Replicas: 101,100,102   Isr: 100,102,101        Elr: N/A        LastKnownElr: N/A
        Topic: test2    Partition: 2    Leader: 102     Replicas: 102,101,100   Isr: 102,101,100        Elr: N/A        LastKnownElr: N/A

0๋ฒˆ ํŒŒํ‹ฐ์…˜์ด ๋ธŒ๋กœ์ปค 102๋ฒˆ์œผ๋กœ ์˜ฎ๊ฒจ์กŒ๋‹ค!! ใ…Žใ…Ž ๊ฐ•์ œ๋กœ ํ•ซ์ŠคํŒŸ์œผ๋กœ ๋งŒ๋“ค์–ด์กŒ์Šต๋‹ˆ๋‹ค ใ…‹ใ…‹

๊ทธ๋Ÿฌ๋‚˜ ์ด๋Ÿฐ ํŒŒํ‹ฐ์…˜ ์žฌ์„ค์ •์€ ์นดํ”„์นด ๋‚ด๋ถ€์— ๋งŽ์€ ์ž‘์—…์„ ์š”๊ตฌํ•˜๊ธฐํ•ด ํด๋Ÿฌ์Šคํ„ฐ์— ๋ถ€ํ•˜๊ฐ€ ์˜ฌ ์ˆ˜ ์žˆ๊ณ , ๋˜ ํ•ด๋‹น ํ† ํ”ฝ์„ ๊ตฌ๋…ํ•˜๋Š” ์ปจ์Šˆ๋จธ์—๊ฒŒ๋„ ๋ฆฌ๋ฐธ๋Ÿฐ์‹ฑ์„ ํŠธ๋ฆฌ๊ฑฐ ํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ์šด์˜ ํ™˜๊ฒฝ์—์„œ๋Š” ์žฌ์„ค์ •์„ ์•„์ฃผ ์กฐ์‹ฌํžˆ ์ง„ํ–‰ํ•ด์•ผ ํ•˜๋ฉฐ, ๋งŒ์•ฝ ์žฅ์• ๊ฐ€ ์˜ˆ์ƒ๋œ๋‹ค๋ฉด ํŒŒํ‹ฐ์…˜์„ ์‹ ๊ทœ ์ƒ์„ฑํ•ด ํ•ด๊ฒฐํ•˜๋Š” ๊ฒƒ์ด ๋ฐฉ๋ฒ•์ผ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

Kafka Delete record

Kafka Dump Logs

Kafka Cluster

kafka-broker-api-versions.sh๋กœ ์นดํ”„์นด ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๊ตฌ์„ฑํ•˜๋Š” ๋ธŒ๋กœ์ปค ๋ชฉ๋ก๊ณผ ๊ฐ ๋ธŒ๋กœ์ปค๊ฐ€ ์ง€์›ํ•˜๋Š” API ๋ฒ„์ „์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

$ kafka-broker-api-versions.sh \
  --bootstrap-server bitnami-kafka:9092
bitnami-kafka-broker-1.bitnami-kafka-broker-headless.kafka.svc.cluster.local:9092 (id: 101 rack: null) -> (
        Produce(0): 0 to 11 [usable: 11],
        Fetch(1): 0 to 17 [usable: 17],
        ListOffsets(2): 0 to 9 [usable: 9],
        Metadata(3): 0 to 12 [usable: 12],
        LeaderAndIsr(4): 0 to 7 [usable: 7],
        StopReplica(5): 0 to 4 [usable: 4],
        UpdateMetadata(6): 0 to 8 [usable: 8],
        ...
)
...

๋ธŒ๋กœ์ปค์˜ ๊ฐ API๋Š” ๊ณ ์œ ํ•œ API Key ๊ฐ’๊ณผ Versioning์ด ๋œ๋‹ค๊ณ  ํ•ฉ๋‹ˆ๋‹ค. ํ˜„์ œ ๋ธŒ๋กœ์ปค์—์„œ ์ง€์›ํ•˜๋Š” API ๋ฒ„์ „์˜ ๋ฒ”์œ„๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Connectors

  • connect-distributed.sh
  • connect-standalone.sh

๊ณ ๊ธ‰

  • kafka-e2e-latency.sh
  • kafka-mirror-maker.sh
  • kafka-streams-application-reset.sh
  • zookeeper-server-start.sh
  • connect-mirror-maker.sh
  • kafka-features.sh
  • kafka-producer-perf-test.sh
  • zookeeper-server-stop.sh
  • connect-plugin-path.sh
  • kafka-get-offsets.sh
  • kafka-reassign-partitions.sh
  • kafka-transactions.sh
  • zookeeper-shell.sh
  • kafka-jmx.sh
  • kafka-replica-verification.sh
  • kafka-verifiable-consumer.sh
  • kafka-acls.sh
  • kafka-consumer-perf-test.sh
  • kafka-leader-election.sh
  • kafka-run-class.sh
  • kafka-verifiable-producer.sh
  • kafka-broker-api-versions.sh
  • kafka-delegation-tokens.sh
  • kafka-log-dirs.sh
  • kafka-server-start.sh
  • trogdor.sh
  • kafka-client-metrics.sh
  • kafka-delete-records.sh
  • kafka-metadata-quorum.sh
  • kafka-server-stop.sh
  • kafka-dump-log.sh
  • kafka-metadata-shell.sh
  • kafka-storage.sh
  • zookeeper-security-migration.sh
  • kafka-cluster.sh

์ฐธ๊ณ ์ž๋ฃŒ

Categories:

Updated: